Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None

      Description

      We have many bundled dependencies in both the source and the binary artifacts that are not in LICENSE.txt and NOTICE.txt.

      1. HADOOP-12893-addendum-branch-2.7.01.patch
        0.8 kB
        Akira Ajisaka
      2. HADOOP-12893.branch-2.7.3.01.patch
        82 kB
        Akira Ajisaka
      3. HADOOP-12893.branch-2.7.02.patch
        82 kB
        Akira Ajisaka
      4. HADOOP-12893.branch-2.7.01.patch
        82 kB
        Akira Ajisaka
      5. HADOOP-12893.branch-2.6.01.patch
        78 kB
        Akira Ajisaka
      6. HADOOP-12893.branch-2.01.patch
        74 kB
        Akira Ajisaka
      7. HADOOP-12893.addendum-trunk.01.patch
        17 kB
        Xiao Chen
      8. HADOOP-12893.10.patch
        71 kB
        Xiao Chen
      9. HADOOP-12893.012.patch
        75 kB
        Akira Ajisaka
      10. HADOOP-12893.011.patch
        72 kB
        Xiao Chen
      11. HADOOP-12893.01.patch
        180 kB
        Xiao Chen
      12. HADOOP-12893.009.patch
        73 kB
        Xiao Chen
      13. HADOOP-12893.008.patch
        73 kB
        Xiao Chen
      14. HADOOP-12893.007.patch
        174 kB
        Andrew Wang
      15. HADOOP-12893.006.patch
        71 kB
        Xiao Chen
      16. HADOOP-12893.005.patch
        186 kB
        Xiao Chen
      17. HADOOP-12893.004.patch
        186 kB
        Xiao Chen
      18. HADOOP-12893.003.patch
        181 kB
        Xiao Chen
      19. HADOOP-12893.002.patch
        181 kB
        Tsuyoshi Ozawa

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reading OpenSSL license.
          https://www.openssl.org/source/license.html

          I'm thinking we need to add the following sentences to NOTICE.txt.

          "This product includes software developed by the OpenSSL Project
          for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
          
          "This product includes cryptographic software written by
          Eric Young (eay@cryptsoft.com)"
          

          I'll read the other following licenses of the softwares in BUILDING.txt

          • protocolbuffer
          • zlib
          • Linux FUSE
          • snappy
          • Intel ISA-L
          • bzip2
          • Jansson
          Show
          ajisakaa Akira Ajisaka added a comment - Reading OpenSSL license. https://www.openssl.org/source/license.html I'm thinking we need to add the following sentences to NOTICE.txt. "This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/)" "This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)" I'll read the other following licenses of the softwares in BUILDING.txt protocolbuffer zlib Linux FUSE snappy Intel ISA-L bzip2 Jansson
          Hide
          aw Allen Wittenauer added a comment -

          All of the bundled/dependent jars need to be gone through as well (e.g., leveldbjni).

          Show
          aw Allen Wittenauer added a comment - All of the bundled/dependent jars need to be gone through as well (e.g., leveldbjni).
          Hide
          aw Allen Wittenauer added a comment -

          ... and all of the javascript, since I see we have bootstrap.

          Show
          aw Allen Wittenauer added a comment - ... and all of the javascript, since I see we have bootstrap.
          Hide
          cmccabe Colin P. McCabe added a comment -

          We do not statically link OpenSSL into anything. It's not part of libhadooppipes.a, and it's not included in libhadoop.so or libhadoop.a. It's not even a depedency of libhadoop, but is accessed via dlopen / dlsym.

          You can see the dependencies of libhadoop.so easily via:

          cmccabe@mirabilis:~/hadoop> ldd ./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop.so
                  linux-vdso.so.1 (0x00007ffdd072f000)
                  libdl.so.2 => /lib64/libdl.so.2 (0x00007f240717d000)
                  libjvm.so => /usr/java/latest/jre/lib/amd64/server/libjvm.so (0x00007f2406457000)
                  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2406239000)
                  libc.so.6 => /lib64/libc.so.6 (0x00007f2405e8b000)
                  /lib64/ld-linux-x86-64.so.2 (0x00007f24075a6000)
                  libm.so.6 => /lib64/libm.so.6 (0x00007f2405b88000)
          

          libdl, libpthread, libc, ld-linux-x86-64.so.2, and libm are system libraries, and libjvm.so comes from the JVM which is installed.

          From http://www.apache.org/dev/licensing-howto.html:

          THE SIMPLE CASE -- NO BUNDLED DEPENDENCIES
          For a source tree which consists entirely of code licensed to the ASF by the copyright holders and which has no bundled dependencies, LICENSE should contain the text of the ALv2 -- no more, no less.
          
          NOTICE should contain only the following text, adapted with the product's name and copyright dates:
          
          Apache [PRODUCT_NAME]
          Copyright [XXXX-20XX] The Apache Software Foundation
          
          This product includes software developed at
          The Apache Software Foundation (http://www.apache.org/).
          

          The way we handle the native dependencies in the LICENSE and NOTICE files right now is correct.

          The javascript libraries are indeed bundled, and should be added to LICENSE and NOTICE.
          find -name '*.js' reveals that we have bundled at least dust.js, moment.js, jquery, bootstrap, and d3.

          Show
          cmccabe Colin P. McCabe added a comment - We do not statically link OpenSSL into anything. It's not part of libhadooppipes.a, and it's not included in libhadoop.so or libhadoop.a. It's not even a depedency of libhadoop, but is accessed via dlopen / dlsym . You can see the dependencies of libhadoop.so easily via: cmccabe@mirabilis:~/hadoop> ldd ./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/ native /libhadoop.so linux-vdso.so.1 (0x00007ffdd072f000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f240717d000) libjvm.so => /usr/java/latest/jre/lib/amd64/server/libjvm.so (0x00007f2406457000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2406239000) libc.so.6 => /lib64/libc.so.6 (0x00007f2405e8b000) /lib64/ld-linux-x86-64.so.2 (0x00007f24075a6000) libm.so.6 => /lib64/libm.so.6 (0x00007f2405b88000) libdl, libpthread, libc, ld-linux-x86-64.so.2, and libm are system libraries, and libjvm.so comes from the JVM which is installed. From http://www.apache.org/dev/licensing-howto.html: THE SIMPLE CASE -- NO BUNDLED DEPENDENCIES For a source tree which consists entirely of code licensed to the ASF by the copyright holders and which has no bundled dependencies, LICENSE should contain the text of the ALv2 -- no more, no less. NOTICE should contain only the following text, adapted with the product's name and copyright dates: Apache [PRODUCT_NAME] Copyright [XXXX-20XX] The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http: //www.apache.org/). The way we handle the native dependencies in the LICENSE and NOTICE files right now is correct. The javascript libraries are indeed bundled, and should be added to LICENSE and NOTICE. find -name '*.js' reveals that we have bundled at least dust.js, moment.js, jquery, bootstrap, and d3.
          Hide
          aw Allen Wittenauer added a comment -

          Colin P. McCabe, I'd appreciate it if you would refrain from changing the JIRA, especially when there is already evidence mentioned prior to your post about other licensing issues beyond just the javascript. Thank you.

          Show
          aw Allen Wittenauer added a comment - Colin P. McCabe , I'd appreciate it if you would refrain from changing the JIRA, especially when there is already evidence mentioned prior to your post about other licensing issues beyond just the javascript. Thank you.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Sorry if my change to the description was unwelcome. I thought it was clear that this was a JS-only issue (but perhaps I was mistaken.) Can you be clearer about what licensing issues you think exist beyond the javascript?

          Show
          cmccabe Colin P. McCabe added a comment - - edited Sorry if my change to the description was unwelcome. I thought it was clear that this was a JS-only issue (but perhaps I was mistaken.) Can you be clearer about what licensing issues you think exist beyond the javascript?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Allen and Colin for comments.

          The way we handle the native dependencies in the LICENSE and NOTICE files right now is correct.

          Agreed.

          Can you be clearer about what licensing issues you think exist beyond the javascript?

          leveldbjni is an example. leveldbjni-all-1.8.jar is included in the binary tarball, so I'm thinking we should add the following license to LICENSE.txt when releasing binary tarball.
          https://github.com/fusesource/leveldbjni/blob/master/license.txt
          http://www.apache.org/dev/licensing-howto.html#binary

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Allen and Colin for comments. The way we handle the native dependencies in the LICENSE and NOTICE files right now is correct. Agreed. Can you be clearer about what licensing issues you think exist beyond the javascript? leveldbjni is an example. leveldbjni-all-1.8.jar is included in the binary tarball, so I'm thinking we should add the following license to LICENSE.txt when releasing binary tarball. https://github.com/fusesource/leveldbjni/blob/master/license.txt http://www.apache.org/dev/licensing-howto.html#binary
          Hide
          aw Allen Wittenauer added a comment - - edited

          Can you be clearer about what licensing issues you think exist beyond the javascript?

          I do not have a complete list, but let's see if we can find one.

          Download https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz . There are jars in binary form in there that we are redistributing. Let's grab one and see what it's license is...

          I know, how about protobuf-java-*.jar? Surely that should be safe given we've been pushing tarballs with it for a few years now, right? With a bit of searching we find this link to the license: https://github.com/google/protobuf/blob/master/LICENSE

          Taking a look at that, we'll find a modified BSD 3-clause license, which contains this text:

          * Redistributions in binary form must reproduce the above
          copyright notice, this list of conditions and the following disclaimer
          in the documentation and/or other materials provided with the
          distribution.
          

          Do you see us reference protobuf's license terms and conditions anywhere in the above tar.gz file? No? Me neither. IANAL, but I'm pretty sure that's a violation of protobuf's license.

          Show
          aw Allen Wittenauer added a comment - - edited Can you be clearer about what licensing issues you think exist beyond the javascript? I do not have a complete list, but let's see if we can find one. Download https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz . There are jars in binary form in there that we are redistributing . Let's grab one and see what it's license is... I know, how about protobuf-java-*.jar? Surely that should be safe given we've been pushing tarballs with it for a few years now, right? With a bit of searching we find this link to the license: https://github.com/google/protobuf/blob/master/LICENSE Taking a look at that, we'll find a modified BSD 3-clause license, which contains this text: * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Do you see us reference protobuf's license terms and conditions anywhere in the above tar.gz file? No? Me neither. IANAL, but I'm pretty sure that's a violation of protobuf's license.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Hmm. Looking at http://www.apache.org/dev/licensing-howto.html again, it seems that maybe we do need to have a separate LICENSE and NOTICE for our binary tarballs, since they have additional bundled contents:

           > BINARY DISTRIBUTIONS
           > What applies to canonical source distributions also applies to all 
           > redistributions, including binary redistributions:
           > 
           > Any redistribution must obey the licensing requirements of the contents.
           > 
           > The best way to do that will likely depend on the binary packaging form.
           > 
           > When assembling binary distributions, it is common to pull in and bundle 
           > additional dependencies which are not bundled with the source 
           > distribution. These additional dependencies must be accounted for in 
           > LICENSE and NOTICE.
          

          I know a few projects that just do source-only releases. It avoids this sort of issue, and also the issues of missing native library support giving a poor end-user experience. I've always wondered why Hadoop didn't go that route.

          I wonder if someone more knowledgeable about licensing can comment on whether our binary tarball does indeed mean we need to make major changes to NOTICE and LICENSE? Those files will be extremely long if we put in all transitive dependencies...

          Show
          cmccabe Colin P. McCabe added a comment - - edited Hmm. Looking at http://www.apache.org/dev/licensing-howto.html again, it seems that maybe we do need to have a separate LICENSE and NOTICE for our binary tarballs, since they have additional bundled contents: > BINARY DISTRIBUTIONS > What applies to canonical source distributions also applies to all > redistributions, including binary redistributions: > > Any redistribution must obey the licensing requirements of the contents. > > The best way to do that will likely depend on the binary packaging form. > > When assembling binary distributions, it is common to pull in and bundle > additional dependencies which are not bundled with the source > distribution. These additional dependencies must be accounted for in > LICENSE and NOTICE. I know a few projects that just do source-only releases. It avoids this sort of issue, and also the issues of missing native library support giving a poor end-user experience. I've always wondered why Hadoop didn't go that route. I wonder if someone more knowledgeable about licensing can comment on whether our binary tarball does indeed mean we need to make major changes to NOTICE and LICENSE? Those files will be extremely long if we put in all transitive dependencies...
          Hide
          aw Allen Wittenauer added a comment -

          Hadoop is big and complex enough that we should probably be using Apache Whisker.

          Show
          aw Allen Wittenauer added a comment - Hadoop is big and complex enough that we should probably be using Apache Whisker.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I tried to use Apache Whisker but I couldn't because of poor documentation and releases. Instead I used license-maven-plugin (http://www.mojohaus.org/license-maven-plugin/), and the output is https://gist.github.com/aajisaka/e602d4e5d7569f5fe32149193c81b749

          Show
          ajisakaa Akira Ajisaka added a comment - I tried to use Apache Whisker but I couldn't because of poor documentation and releases. Instead I used license-maven-plugin ( http://www.mojohaus.org/license-maven-plugin/ ), and the output is https://gist.github.com/aajisaka/e602d4e5d7569f5fe32149193c81b749
          Hide
          busbey Sean Busbey added a comment -

          copying my comments from common-dev@

          Each artifact that the PMC publishes must abide by the ASF licensing
          policy. That includes

          • Source release artifact
          • any convenience binary artifacts places on dist.apache
          • any convenience jars put into the ASF Nexus repository

          That likely means that we bundle much more than just what's in the source tree.

          So we might end up needing different LICENSE/NOTICE entries for source and binary tarballs and for some of our jars. They do end up being very large (I could attach an example from HBase if folks like, or NiFi, if it would help to see what projects that have spent a fair bit of time on this ended up doing). One option is to have a directory with various license files and then reference them in our LICENSE file. There's no such shortcut available for things that require a NOTICE entry.

          In HBase this took a long time to get right and we largely had to do it by manually reviewing every artifact and leveraging the assembly and remote-resources plugins. I had wanted to use Apache Whisker, but I ran into the same kind of problems as Akira Ajisaka mentioned.

          Show
          busbey Sean Busbey added a comment - copying my comments from common-dev@ Each artifact that the PMC publishes must abide by the ASF licensing policy. That includes Source release artifact any convenience binary artifacts places on dist.apache any convenience jars put into the ASF Nexus repository That likely means that we bundle much more than just what's in the source tree. So we might end up needing different LICENSE/NOTICE entries for source and binary tarballs and for some of our jars. They do end up being very large (I could attach an example from HBase if folks like, or NiFi, if it would help to see what projects that have spent a fair bit of time on this ended up doing). One option is to have a directory with various license files and then reference them in our LICENSE file. There's no such shortcut available for things that require a NOTICE entry. In HBase this took a long time to get right and we largely had to do it by manually reviewing every artifact and leveraging the assembly and remote-resources plugins. I had wanted to use Apache Whisker, but I ran into the same kind of problems as Akira Ajisaka mentioned.
          Hide
          billie.rinaldi Billie Rinaldi added a comment -

          I looked into L&N changes needed for leveldbjni-all-1.8.jar in YARN-1704. The files that I changed no longer exist, so I'm not sure what happened to those, but someone could reuse the content.

          Show
          billie.rinaldi Billie Rinaldi added a comment - I looked into L&N changes needed for leveldbjni-all-1.8.jar in YARN-1704 . The files that I changed no longer exist, so I'm not sure what happened to those, but someone could reuse the content.
          Hide
          aw Allen Wittenauer added a comment -

          When HADOOP-10956 was added, the contents of YARN's custom LICENSE.txt and NOTICE.txt weren't merged into the master one. Effectively another victim of the "oh one day we'll split the sub-projects" madness that sweeps through the PMC every-so-often since we had four of these files instead of just one.

          Show
          aw Allen Wittenauer added a comment - When HADOOP-10956 was added, the contents of YARN's custom LICENSE.txt and NOTICE.txt weren't merged into the master one. Effectively another victim of the "oh one day we'll split the sub-projects" madness that sweeps through the PMC every-so-often since we had four of these files instead of just one.
          Hide
          andrew.wang Andrew Wang added a comment -

          I posted a patch at HADOOP-13042 to restore the leveldbjni-related LICENSE and NOTICE txt information lost in HADOOP-10956, also audited that change to make sure nothing else was lost.

          Show
          andrew.wang Andrew Wang added a comment - I posted a patch at HADOOP-13042 to restore the leveldbjni-related LICENSE and NOTICE txt information lost in HADOOP-10956 , also audited that change to make sure nothing else was lost.
          Hide
          andrew.wang Andrew Wang added a comment -

          I also put up HADOOP-13043 for the javascript stuff. Once this goes in, I think at least our source releases are in the clear.

          Looking at Akira's list of maven dependencies, there's a lot. Even for the ASF projects, we still need to check their NOTICE files.

          Help would definitely be appreciated here. I'll get started on my own, but ping me (andrew.wang@cloudera.com) if you're interested and we can collaborate on a Google Doc.

          Show
          andrew.wang Andrew Wang added a comment - I also put up HADOOP-13043 for the javascript stuff. Once this goes in, I think at least our source releases are in the clear. Looking at Akira's list of maven dependencies, there's a lot. Even for the ASF projects, we still need to check their NOTICE files. Help would definitely be appreciated here. I'll get started on my own, but ping me (andrew.wang@cloudera.com) if you're interested and we can collaborate on a Google Doc.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Andrew Wang for starting to work on this!

          General comments/questions about LICENSE/NOTICE changes:
          1) I'm not sure if it is discussed: I noticed HADOOP-13042 updated NOTICE.txt. However, http://www.apache.org/dev/licensing-howto.html#mod-notice mentioned that

          ... Aside from Apache-licensed dependencies which supply NOTICE files of their own, it is uncommon for a dependency to require additions to NOTICE ...

          And

          ... It is important to keep NOTICE as brief and simple as possible, as each addition places a burden on downstream consumers. ...

          Is there any guideline about when NOTICE has to be updated?

          2) Filed a legal question LEGAL-247 about if we can merge dependencies which have same license. LICENSE.txt can be simplified a lot if the merging is allowed.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Andrew Wang for starting to work on this! General comments/questions about LICENSE/NOTICE changes: 1) I'm not sure if it is discussed: I noticed HADOOP-13042 updated NOTICE.txt. However, http://www.apache.org/dev/licensing-howto.html#mod-notice mentioned that ... Aside from Apache-licensed dependencies which supply NOTICE files of their own, it is uncommon for a dependency to require additions to NOTICE ... And ... It is important to keep NOTICE as brief and simple as possible, as each addition places a burden on downstream consumers. ... Is there any guideline about when NOTICE has to be updated? 2) Filed a legal question LEGAL-247 about if we can merge dependencies which have same license. LICENSE.txt can be simplified a lot if the merging is allowed.
          Hide
          andrew.wang Andrew Wang added a comment -

          Good point Wangda. http://www.apache.org/dev/licensing-howto.html has this statement:

          However, elements such as the copyright notifications embedded within BSD and MIT licenses need not be duplicated in NOTICE – it suffices to leave those notices in their original locations.

          The original YARN-1704 added these NOTICE changes, but it seems like we can probably drop them since it's all 3-clause BSD. There's discussion in the linked LEGAL-59 about this case.

          Show
          andrew.wang Andrew Wang added a comment - Good point Wangda. http://www.apache.org/dev/licensing-howto.html has this statement: However, elements such as the copyright notifications embedded within BSD and MIT licenses need not be duplicated in NOTICE – it suffices to leave those notices in their original locations. The original YARN-1704 added these NOTICE changes, but it seems like we can probably drop them since it's all 3-clause BSD. There's discussion in the linked LEGAL-59 about this case.
          Hide
          andrew.wang Andrew Wang added a comment -

          Actually, we missed the obvious: the LevelDB stff is the exact case of an Apache-licensed dependency with its own NOTICE files. Should have read YARN-1704 closer

          Show
          andrew.wang Andrew Wang added a comment - Actually, we missed the obvious: the LevelDB stff is the exact case of an Apache-licensed dependency with its own NOTICE files. Should have read YARN-1704 closer
          Hide
          leftnoteasy Wangda Tan added a comment -

          Andrew Wang,

          Agree, yes we need them

          Show
          leftnoteasy Wangda Tan added a comment - Andrew Wang , Agree, yes we need them
          Hide
          andrew.wang Andrew Wang added a comment -

          I got both my above patches in, thanks Steve for reviewing.

          LEGAL-247 was resolved in favor of merging licenses, meaning we fortunately won't need a zillion copies of BSD/ALv2/etc. I think the overall goal here is to make our best effort at being honest about our dependencies. I surveyed a few different Apache projects, and there's no one true style for LICENSE and NOTICE.

          Akira Ajisaka and Xiao Chen generously volunteered to help with this. I created a Google spreadsheet that we can use to hopefully auto-generate the LICENSE/NOTICE files later on. Ping me if you want access to that.

          Besides that, I could also use some help to figure out how to copy the LICENSE and NOTICE into our JAR files. HBase does this (related JIRA HBASE-14085), so we might be able to reuse their pom work.

          Show
          andrew.wang Andrew Wang added a comment - I got both my above patches in, thanks Steve for reviewing. LEGAL-247 was resolved in favor of merging licenses, meaning we fortunately won't need a zillion copies of BSD/ALv2/etc. I think the overall goal here is to make our best effort at being honest about our dependencies. I surveyed a few different Apache projects, and there's no one true style for LICENSE and NOTICE. Akira Ajisaka and Xiao Chen generously volunteered to help with this. I created a Google spreadsheet that we can use to hopefully auto-generate the LICENSE/NOTICE files later on. Ping me if you want access to that. Besides that, I could also use some help to figure out how to copy the LICENSE and NOTICE into our JAR files. HBase does this (related JIRA HBASE-14085 ), so we might be able to reuse their pom work.
          Hide
          xiaochen Xiao Chen added a comment -

          Update:
          So far we've gotten a consolidated list of LICENSE and NOTICE. Will be working on finalizing it, and also try to copy it into our JAR files.

          Thanks Andrew Wang and Akira Ajisaka for working on this together!

          Show
          xiaochen Xiao Chen added a comment - Update: So far we've gotten a consolidated list of LICENSE and NOTICE. Will be working on finalizing it, and also try to copy it into our JAR files. Thanks Andrew Wang and Akira Ajisaka for working on this together!
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Xiao Chen, Andrew Wang and Akira Ajisaka for working on this issue!

          Is there anything more needed for LICENSE.txt/NOTICE.txt updating?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Xiao Chen , Andrew Wang and Akira Ajisaka for working on this issue! Is there anything more needed for LICENSE.txt/NOTICE.txt updating?
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Tan, Wangda for offering help! I was blocked by jira...
          Attached a patch 1 for the work against trunk.
          I think for now the most helpful thing is to review this patch and have trunk done. After that I guess we'll need to work on every release branches.

          The way we (Andrew Wang + Akira Ajisaka + myself) did this:

          1. Have whisker to generate all dependencies, consolidate it into a spreadsheet.
          2. Manually find license/notice for each of them. Same license for different dependencies are merged per LEGAL-247.
          3. Parse the result into new LICENSE and NOTICE files
          4. Manually add merge the new L&N into current ones
          5. Add a new module hadoop-resource-bundle to easily patch L&N into jars' META-INF section. (Thank you HBase for showing an example)
          6. Verify that all jars contain the L&N by grepping the generated jars. e.g. for f in $(find ./target/hadoop-3.0.0-alpha1-SNAPSHOT/share -name hadoop*jar); do echo $f;jar -tf $f|grep "LICENSE|NOTICE";done
          Show
          xiaochen Xiao Chen added a comment - Thanks Tan, Wangda for offering help! I was blocked by jira... Attached a patch 1 for the work against trunk. I think for now the most helpful thing is to review this patch and have trunk done. After that I guess we'll need to work on every release branches. The way we ( Andrew Wang + Akira Ajisaka + myself) did this: Have whisker to generate all dependencies, consolidate it into a spreadsheet. Manually find license/notice for each of them. Same license for different dependencies are merged per LEGAL-247 . Parse the result into new LICENSE and NOTICE files Manually add merge the new L&N into current ones Add a new module hadoop-resource-bundle to easily patch L&N into jars' META-INF section. (Thank you HBase for showing an example) Verify that all jars contain the L&N by grepping the generated jars. e.g. for f in $(find ./target/hadoop-3.0.0-alpha1-SNAPSHOT/share -name hadoop*jar); do echo $f;jar -tf $f|grep "LICENSE|NOTICE";done
          Hide
          busbey Sean Busbey added a comment -

          An important note: it looks like y'all are planning on a single LICENSE/NOTICE pair for all artifacts. ASF policy requires that the LICENSE/NOTICE file cover exactly the contents of each distributed artifact. That likely means that we need different ones for hte source tarball, the convenience binary tarball, and different ones depending on the specific contents of jar files (not just per-module, but for -source.jar, -javadoc.jar, -test.jar).

          Show
          busbey Sean Busbey added a comment - An important note: it looks like y'all are planning on a single LICENSE/NOTICE pair for all artifacts. ASF policy requires that the LICENSE/NOTICE file cover exactly the contents of each distributed artifact. That likely means that we need different ones for hte source tarball, the convenience binary tarball, and different ones depending on the specific contents of jar files (not just per-module, but for -source.jar, -javadoc.jar, -test.jar).
          Hide
          leftnoteasy Wangda Tan added a comment -

          Xiao Chen, Andrew Wang and Akira Ajisaka, thanks a lot for the great work!

          I can help with reviewing this patch.

          ASF policy requires that the LICENSE/NOTICE file cover exactly the contents of each distributed artifact.

          My understanding is LICENSE / NOTICE of binary distribution should be a superset of source distribution. Is it good enough to have a separate binary-distribution-only LICENSE / NOTICE file and we can concat binary-distribution-only and source-distribution L/N while releasing?

          Show
          leftnoteasy Wangda Tan added a comment - Xiao Chen , Andrew Wang and Akira Ajisaka , thanks a lot for the great work! I can help with reviewing this patch. ASF policy requires that the LICENSE/NOTICE file cover exactly the contents of each distributed artifact. My understanding is LICENSE / NOTICE of binary distribution should be a superset of source distribution. Is it good enough to have a separate binary-distribution-only LICENSE / NOTICE file and we can concat binary-distribution-only and source-distribution L/N while releasing?
          Hide
          andrew.wang Andrew Wang added a comment -

          In the L&N we say whether something applies to the binary or the source
          distribution. I saw this elsewhere, and it really reduces the POM work
          required.

          I'd like to do something similar for our JARs. We can list out which JARs
          are affected by each L&N entry. Fortunately we only need to do this for the
          source L&Ns, which are not too numerous.

          I'd like to appeal to a reasonable person standard. We're making a big
          effort here to be compliant, and if we do the above, it'll be clear what
          does and doesn't apply to each artifact. In the meanwhile, our releases are
          blocked.

          If additional work really is required, maybe it could also be done as a
          follow-on.

          Show
          andrew.wang Andrew Wang added a comment - In the L&N we say whether something applies to the binary or the source distribution. I saw this elsewhere, and it really reduces the POM work required. I'd like to do something similar for our JARs. We can list out which JARs are affected by each L&N entry. Fortunately we only need to do this for the source L&Ns, which are not too numerous. I'd like to appeal to a reasonable person standard. We're making a big effort here to be compliant, and if we do the above, it'll be clear what does and doesn't apply to each artifact. In the meanwhile, our releases are blocked. If additional work really is required, maybe it could also be done as a follow-on.
          Hide
          busbey Sean Busbey added a comment -

          My understanding is LICENSE / NOTICE of binary distribution should be a superset of source distribution. Is it good enough to have a separate binary-distribution-only LICENSE / NOTICE file and we can concat binary-distribution-only and source-distribution L/N while releasing?

          This is not necessarily true, though I haven't done a sufficient review to say if it is for Hadoop or not. As an example, one could have some third party code bundled in the test sources and produce a binary distribution tarball with no test files in it. Similarly, if the main classes include some third party work but the tests do not, then the main jar and the test jar would be different. (which would matter if the test jar is published to maven.)

          In the L&N we say whether something applies to the binary or the source
          distribution. I saw this elsewhere, and it really reduces the POM work
          required.

          I've seen this a few places, but unfortunately it's incorrect. I've been slowly working through projects to help correct them, but it's a long slog.

          I'd like to appeal to a reasonable person standard. We're making a big
          effort here to be compliant, and if we do the above, it'll be clear what
          does and doesn't apply to each artifact. In the meanwhile, our releases are
          blocked.

          If additional work really is required, maybe it could also be done as a
          follow-on.

          That's entirely up to the Hadoop PMC. I can certainly understand the reasoning of an incremental approach that starts with getting us out of violating the licenses of third parties and works towards compliance with ASF Policy.

          I would be concerned if "follow-on" turned into "next release" perpetually; having releases blocked provides a kind of motivation that little else can. We need to end up in a place where everything we distribute meets ASF Policy, but folks generally understand that this can take some time.

          Keep in mind that release voting is majority, so it might be worth a straw poll of how the PMC would vote if a given release met the requirements for third party licenses but did not yet meet ASF policy on license notifications.

          Show
          busbey Sean Busbey added a comment - My understanding is LICENSE / NOTICE of binary distribution should be a superset of source distribution. Is it good enough to have a separate binary-distribution-only LICENSE / NOTICE file and we can concat binary-distribution-only and source-distribution L/N while releasing? This is not necessarily true, though I haven't done a sufficient review to say if it is for Hadoop or not. As an example, one could have some third party code bundled in the test sources and produce a binary distribution tarball with no test files in it. Similarly, if the main classes include some third party work but the tests do not, then the main jar and the test jar would be different. (which would matter if the test jar is published to maven.) In the L&N we say whether something applies to the binary or the source distribution. I saw this elsewhere, and it really reduces the POM work required. I've seen this a few places, but unfortunately it's incorrect. I've been slowly working through projects to help correct them, but it's a long slog. I'd like to appeal to a reasonable person standard. We're making a big effort here to be compliant, and if we do the above, it'll be clear what does and doesn't apply to each artifact. In the meanwhile, our releases are blocked. If additional work really is required, maybe it could also be done as a follow-on. That's entirely up to the Hadoop PMC. I can certainly understand the reasoning of an incremental approach that starts with getting us out of violating the licenses of third parties and works towards compliance with ASF Policy. I would be concerned if "follow-on" turned into "next release" perpetually; having releases blocked provides a kind of motivation that little else can. We need to end up in a place where everything we distribute meets ASF Policy, but folks generally understand that this can take some time. Keep in mind that release voting is majority, so it might be worth a straw poll of how the PMC would vote if a given release met the requirements for third party licenses but did not yet meet ASF policy on license notifications.
          Hide
          andrew.wang Andrew Wang added a comment - - edited

          Thanks Sean Busbey. I sent out a mail to common-dev, we'll see about what comes back. Do you have references for these statements? I believe you, but I would like a comprehensive guide for what's required. Else we're liable to still get it wrong.

          In the meanwhile, some review comments based on the Apache page on this: http://www.apache.org/dev/licensing-howto.html

          Show
          andrew.wang Andrew Wang added a comment - - edited Thanks Sean Busbey . I sent out a mail to common-dev, we'll see about what comes back. Do you have references for these statements? I believe you, but I would like a comprehensive guide for what's required. Else we're liable to still get it wrong. In the meanwhile, some review comments based on the Apache page on this: http://www.apache.org/dev/licensing-howto.html No need to list ALv2 dependencies in LICENSE We're missing version numbers. Licenses can change in later releases. IIUC we don't need to list the copyrights for BSD and MIT licensed deps in NOTICE . I'm not 100% sure how this applies to included JARs, but the understanding for source files is that you just leave their copyright headers and it's okay. Included JARs should also have their LICENSE/NOTICE in their own META-INF dirs, which would also satisfy the copyright requirement. I'd also be in favor of checking the source data and scripts into the repo, to make this easier in the future. Ideally all these files are auto-generated.
          Hide
          busbey Sean Busbey added a comment -

          Thanks Sean Busbey. I sent out a mail to common-dev, we'll see about what comes back. Do you have references for these statements? I believe you, but I would like a comprehensive guide for what's required. Else we're liable to still get it wrong.

          Sure. I'm guessing these are the two statements in question, if there was another just let me know.

          IIUC we don't need to list the copyrights for BSD and MIT licensed deps in NOTICE. I'm not 100% sure how this applies to included JARs, but the understanding for source files is that you just leave their copyright headers and it's okay. Included JARs should also have their LICENSE/NOTICE in their own META-INF dirs, which would also satisfy the copyright requirement.

          That's correct, you should not include anything from BSD or MIT licensed deps in NOTICE. That includes bundled JARs; in both source and bundled binary cases you should have a reference in the LICENSE file for the included work. (ref licensing howto on permissive licenses)

          Show
          busbey Sean Busbey added a comment - Thanks Sean Busbey. I sent out a mail to common-dev, we'll see about what comes back. Do you have references for these statements? I believe you, but I would like a comprehensive guide for what's required. Else we're liable to still get it wrong. Sure. I'm guessing these are the two statements in question, if there was another just let me know. LICENSE/NOTICE files should cover just the artifact in question gets covered by the ASF Release Policy section on licensing Release votes have to be Majority gets covered by the ASF Release Policy section on approval IIUC we don't need to list the copyrights for BSD and MIT licensed deps in NOTICE. I'm not 100% sure how this applies to included JARs, but the understanding for source files is that you just leave their copyright headers and it's okay. Included JARs should also have their LICENSE/NOTICE in their own META-INF dirs, which would also satisfy the copyright requirement. That's correct, you should not include anything from BSD or MIT licensed deps in NOTICE. That includes bundled JARs; in both source and bundled binary cases you should have a reference in the LICENSE file for the included work. (ref licensing howto on permissive licenses )
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Sean, that's the page I hadn't read yet I really wish the Licensing HOWTO gave examples for how this applies to a typical Java project. It talks a lot about source releases, which are the least common way for people to consume our artifacts. For a long time we were also under the impression that we could just provide the binary artifacts as a "convenience" while the source tarball was the only "official" release artifact. Now we know that's definitely not okay.

          I'm going to fix up the patch based on my above comments. So far no one has raised any objections on the dev list, so we'll have the "good faith" level of compliance to unblock the releases. I'll also attempt the pom work for generating specific L&Ns, but no promises.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Sean, that's the page I hadn't read yet I really wish the Licensing HOWTO gave examples for how this applies to a typical Java project. It talks a lot about source releases, which are the least common way for people to consume our artifacts. For a long time we were also under the impression that we could just provide the binary artifacts as a "convenience" while the source tarball was the only "official" release artifact. Now we know that's definitely not okay. I'm going to fix up the patch based on my above comments. So far no one has raised any objections on the dev list, so we'll have the "good faith" level of compliance to unblock the releases. I'll also attempt the pom work for generating specific L&Ns, but no promises.
          Hide
          billie.rinaldi Billie Rinaldi added a comment -

          I took a look at the patch. Apache releases cannot contain any LGPL dependencies, so if any exist they must be removed before a release can be made. Going through the list of things that are stated to be LGPL in this patch:

          The following items I can't find bundled; can anyone else? If they aren't actually bundled, we don't need to mention them.

          I haven't looked in detail at the other additions in the patch, but we should make sure they are all bundled dependencies. I was wondering about the inclusion of mockito and junit, for example. Are those actually needed, or is it just a case of not specifying the test scope for those dependencies in some of the modules?

          Show
          billie.rinaldi Billie Rinaldi added a comment - I took a look at the patch. Apache releases cannot contain any LGPL dependencies, so if any exist they must be removed before a release can be made. Going through the list of things that are stated to be LGPL in this patch: FindBugs-jsr305 looks like it might be BSD, but I'm not sure ( https://github.com/findbugsproject/findbugs/blob/master/findbugs/licenses/LICENSE-jsr305.txt ) Data Mapper for Jackson and Xml Compatibility extensions for Jackson are dual-licensed AL ( http://wiki.fasterxml.com/JacksonDownload under "Licensing") Javassist is triple licensed MPL/LGPL/Apache ( https://github.com/jboss-javassist/javassist/tree/rel_3_18_1_ga ) The following items I can't find bundled; can anyone else? If they aren't actually bundled, we don't need to mention them. logback could be licensed under EPL instead ( http://logback.qos.ch/license.html ) jdiff is LGPL and can't be bundled I haven't looked in detail at the other additions in the patch, but we should make sure they are all bundled dependencies. I was wondering about the inclusion of mockito and junit, for example. Are those actually needed, or is it just a case of not specifying the test scope for those dependencies in some of the modules?
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm working through the above, thanks for the review Billie.

          I've been looking at HBase's NOTICE as an example. Sean Busbey, hope you don't mind if I ask some questions?

          • The Hadoop NOTICE entry. The guidelines say you don't need to duplicate this normally?
          • The Bootstrap and Guava NOTICE entries. They are both ASLv2 and don't have their own NOTICE files. Are these notices required?

          I ask these because the guidelines say: "Do not add anything to NOTICE which is not legally required."

          So far I've seen a huge variety of NOTICE styles looking through Hadoop's deps, and I'm still looking for a canonical example of what this is supposed to look like.

          Show
          andrew.wang Andrew Wang added a comment - I'm working through the above, thanks for the review Billie. I've been looking at HBase's NOTICE as an example. Sean Busbey , hope you don't mind if I ask some questions? The Hadoop NOTICE entry. The guidelines say you don't need to duplicate this normally? The Bootstrap and Guava NOTICE entries. They are both ASLv2 and don't have their own NOTICE files. Are these notices required? I ask these because the guidelines say: "Do not add anything to NOTICE which is not legally required." So far I've seen a huge variety of NOTICE styles looking through Hadoop's deps, and I'm still looking for a canonical example of what this is supposed to look like.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I haven't looked in detail at the other additions in the patch, but we should make sure they are all bundled dependencies.

          I created a gist to list all the bundled dependencies. Hope it helps.

          Show
          ajisakaa Akira Ajisaka added a comment - I haven't looked in detail at the other additions in the patch, but we should make sure they are all bundled dependencies. I created a gist to list all the bundled dependencies. Hope it helps.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          I will check the jar side today.

          Show
          ozawa Tsuyoshi Ozawa added a comment - I will check the jar side today.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Tsuyoshi Ozawa for helping. I've added you to the spreadsheet that we're co-editing. Hope this can reduce some overhead.

          Show
          xiaochen Xiao Chen added a comment - Thanks Tsuyoshi Ozawa for helping. I've added you to the spreadsheet that we're co-editing. Hope this can reduce some overhead.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Xiao Chen thanks a lot! Taking a look.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Xiao Chen thanks a lot! Taking a look.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          I'm working to update a patch to remove jdiff from jar based on Xiao's spread sheet. jdiff is binary which is incompatible with ASLv2.

          It's from hadoop-annotation. Adding a link to the origin jira on which jdiff is introduced to jar.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited I'm working to update a patch to remove jdiff from jar based on Xiao's spread sheet. jdiff is binary which is incompatible with ASLv2. It's from hadoop-annotation. Adding a link to the origin jira on which jdiff is introduced to jar.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Attaching a patch to make the scope of jdiff to "compile", since it only called when maven profile "-Pdoc" is provided.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Attaching a patch to make the scope of jdiff to "compile", since it only called when maven profile "-Pdoc" is provided.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Xiao Chen Andrew Wang I'd like to test the patch, but I cannot run mvn install correctly with the patch. Do you know how to run it?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Xiao Chen Andrew Wang I'd like to test the patch, but I cannot run mvn install correctly with the patch. Do you know how to run it?
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Tsuyoshi Ozawa! What error do you get?
          I just tried patch 02 on latest trunk (42c22f7e3d6e88bf1115f617f6e803288886d1ac), mvn clean install -DskipTests -T4 -Dmaven.javadoc.skip=true and mvn package -Dtar -Pdist -DskipTests -Dmaven.javadoc.skip=true both succeeded. Trunk was switched to build jdk8 only recently BTW.

          Show
          xiaochen Xiao Chen added a comment - Thanks Tsuyoshi Ozawa ! What error do you get? I just tried patch 02 on latest trunk (42c22f7e3d6e88bf1115f617f6e803288886d1ac), mvn clean install -DskipTests -T4 -Dmaven.javadoc.skip=true and mvn package -Dtar -Pdist -DskipTests -Dmaven.javadoc.skip=true both succeeded. Trunk was switched to build jdk8 only recently BTW.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Akira Ajisaka and Tsuyoshi Ozawa for the work together! We now can distinguish between dependencies of source v.s. bundled. The script is also updated to generate bundled and source L&N separately.

          We should be able to integrate with Andrew Wang's pom work when he comes back on Tuesday.

          Show
          xiaochen Xiao Chen added a comment - Thanks Akira Ajisaka and Tsuyoshi Ozawa for the work together! We now can distinguish between dependencies of source v.s. bundled. The script is also updated to generate bundled and source L&N separately. We should be able to integrate with Andrew Wang 's pom work when he comes back on Tuesday.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Xiao Chen I tried following commands:

          <on trunk>
          $ git apply -p0  HADOOP-12893.002.patch
          $ mvn clean install -DskipTests
          

          Result is as follows:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hadoop-project: Resources archive cannot be found. Could not find artifact org.apache.hadoop:hadoop-resource-bundle:jar:3.0.0-alpha1-SNAPSHOT
          [ERROR]
          [ERROR] Try downloading the file manually from the project website.
          [ERROR]
          [ERROR] Then, install it using the command:
          [ERROR] mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-resource-bundle -Dversion=3.0.0-alpha1-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
          [ERROR]
          [ERROR] Alternatively, if you host your own repository you can deploy the file there:
          [ERROR] mvn deploy:deploy-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-resource-bundle -Dversion=3.0.0-alpha1-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=url -DrepositoryId=[id]
          [ERROR]
          [ERROR]
          [ERROR] org.apache.hadoop:hadoop-resource-bundle:jar:3.0.0-alpha1-SNAPSHOT
          [ERROR]
          [ERROR] from the specified remote repositories:
          [ERROR] apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots, releases=true, snapshots=true),
          [ERROR] repository.jboss.org (http://repository.jboss.org/nexus/content/groups/public/, releases=true, snapshots=false),
          [ERROR] central (http://repo.maven.apache.org/maven2, releases=true, snapshots=false)
          [ERROR] -> [Help 1]
          [ERROR]

          It seems that license META-INF looks to be missing on server. Hence, I also tried to install following command, but it still failed.

          mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-resource-bundle -Dversion=3.0.0-alpha1-SNAPSHOT -Dpackaging=jar -Dfile=./hadoop-resource-bundle/src/main/resources/META-INF/*.txt
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - Xiao Chen I tried following commands: <on trunk> $ git apply -p0 HADOOP-12893.002.patch $ mvn clean install -DskipTests Result is as follows: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hadoop-project: Resources archive cannot be found. Could not find artifact org.apache.hadoop:hadoop-resource-bundle:jar:3.0.0-alpha1-SNAPSHOT [ERROR] [ERROR] Try downloading the file manually from the project website. [ERROR] [ERROR] Then, install it using the command: [ERROR] mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-resource-bundle -Dversion=3.0.0-alpha1-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file [ERROR] [ERROR] Alternatively, if you host your own repository you can deploy the file there: [ERROR] mvn deploy:deploy-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-resource-bundle -Dversion=3.0.0-alpha1-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=url -DrepositoryId=[id] [ERROR] [ERROR] [ERROR] org.apache.hadoop:hadoop-resource-bundle:jar:3.0.0-alpha1-SNAPSHOT [ERROR] [ERROR] from the specified remote repositories: [ERROR] apache.snapshots.https ( https://repository.apache.org/content/repositories/snapshots , releases=true, snapshots=true), [ERROR] repository.jboss.org ( http://repository.jboss.org/nexus/content/groups/public/ , releases=true, snapshots=false), [ERROR] central ( http://repo.maven.apache.org/maven2 , releases=true, snapshots=false) [ERROR] -> [Help 1] [ERROR] It seems that license META-INF looks to be missing on server. Hence, I also tried to install following command, but it still failed. mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-resource-bundle -Dversion=3.0.0-alpha1-SNAPSHOT -Dpackaging=jar -Dfile=./hadoop-resource-bundle/src/main/resources/META-INF/*.txt
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Tsuyoshi Ozawa, I doubt that may has to do with the symlink I used for the L&N files. I'm attaching patch 3 which copies the files instead of having a symlink of them. Compiled and verified locally. Could you try it out?

          Didn't update the content of L&N from patch 1 here, since we need to separate them for source/bundle etc. soon.

          Show
          xiaochen Xiao Chen added a comment - Thanks Tsuyoshi Ozawa , I doubt that may has to do with the symlink I used for the L&N files. I'm attaching patch 3 which copies the files instead of having a symlink of them. Compiled and verified locally. Could you try it out? Didn't update the content of L&N from patch 1 here, since we need to separate them for source/bundle etc. soon.
          Hide
          andrew.wang Andrew Wang added a comment -

          I don't have any new pom work to report, since it seemed like the PMC is okay with the "good faith" solution for now. I think the idea is we refine the specificity of the L&N in a follow-on JIRA. To just get the releases unblocked, we use the same mega-L&N for all our deps.

          I spent my time going through our deps to correct the NOTICE entries and select the more permissive license when available. Assuming I interpreted the Apache requirements correctly, we should be set there.

          If we've successfully excised jdiff, I think we're getting close. It'd be good to also fix the dependency scope of the test-related deps that Billie found, since it's unlikely we need to bundle junit and mockito.

          Show
          andrew.wang Andrew Wang added a comment - I don't have any new pom work to report, since it seemed like the PMC is okay with the "good faith" solution for now. I think the idea is we refine the specificity of the L&N in a follow-on JIRA. To just get the releases unblocked, we use the same mega-L&N for all our deps. I spent my time going through our deps to correct the NOTICE entries and select the more permissive license when available. Assuming I interpreted the Apache requirements correctly, we should be set there. If we've successfully excised jdiff, I think we're getting close. It'd be good to also fix the dependency scope of the test-related deps that Billie found, since it's unlikely we need to bundle junit and mockito.
          Hide
          aw Allen Wittenauer added a comment -

          junit is covered here: HADOOP-8738

          Show
          aw Allen Wittenauer added a comment - junit is covered here: HADOOP-8738
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks a lot Andrew Wang for responding from you vacation! I went ahead and updated the L&Ns from our co-edited spreadsheet.

          It fixes:

          • From Andrew's comment:
            • Not listing ALv2 dependencies in LICENSE
            • Add version numbers to dependencies
            • No copyrights for BSD / MIT licensed deps in NOTICE
            • For the scripts, since we have it in the spreadsheet now, I'm thinking maybe we can make it viewable to all?
            • In the L&N we say whether something applies to the binary or the source
              distribution. I saw this elsewhere, and it really reduces the POM work
              required.

              This is done in this patch, I mentioned in LICENSE that it applies to source or binary (or both). Looking at the NOTICE, it seems already saying 'The binary distribution of this product bundles binaries of'..., so I think we're good using binary only.

            • Reflected the offline idea of merging components under same dependency, to reduce the list.
          • From Bill's comment:
            • Dual-licensed/Tri-licensed deps are now listed under more permissive license (ASLv2 > others)
            • junit and mockito are currently bundled, and I'm removing them from being bundled in this patch. Also thanks Allen for the pointer jira.
            • jdiff is not bundled. But it is used in source (in hadoop-project/pom.xml). I'm not sure how to proceed with this one... Shall we 1) try remove it completely 2) as-is 3) replace it with something else? We listed jdiff in LICENSE for source distribution (for honesty...)
            • Unfortunately we don't have a way to automatically check whether each dependency should be bundled. So if anyone sees another dependency shouldn't be bundled, please feel free to comment.

          Thank you again to everyone for the review and feedback.

          Show
          xiaochen Xiao Chen added a comment - Thanks a lot Andrew Wang for responding from you vacation! I went ahead and updated the L&Ns from our co-edited spreadsheet. It fixes: From Andrew's comment : Not listing ALv2 dependencies in LICENSE Add version numbers to dependencies No copyrights for BSD / MIT licensed deps in NOTICE For the scripts, since we have it in the spreadsheet now, I'm thinking maybe we can make it viewable to all? In the L&N we say whether something applies to the binary or the source distribution. I saw this elsewhere, and it really reduces the POM work required. This is done in this patch, I mentioned in LICENSE that it applies to source or binary (or both). Looking at the NOTICE, it seems already saying 'The binary distribution of this product bundles binaries of'..., so I think we're good using binary only. Reflected the offline idea of merging components under same dependency, to reduce the list. From Bill's comment : Dual-licensed/Tri-licensed deps are now listed under more permissive license (ASLv2 > others) junit and mockito are currently bundled, and I'm removing them from being bundled in this patch. Also thanks Allen for the pointer jira. jdiff is not bundled. But it is used in source (in hadoop-project/pom.xml ). I'm not sure how to proceed with this one... Shall we 1) try remove it completely 2) as-is 3) replace it with something else? We listed jdiff in LICENSE for source distribution (for honesty...) Unfortunately we don't have a way to automatically check whether each dependency should be bundled. So if anyone sees another dependency shouldn't be bundled, please feel free to comment. Thank you again to everyone for the review and feedback.
          Hide
          aw Allen Wittenauer added a comment -

          junit and mockito are currently bundled, and I'm removing them from being bundled in this patch. Also thanks Allen for the pointer jira.

          I don't think you fully understand what's happening in that other JIRA. Removal of junit was specifically reverted because:

          This broke the mapreduce jobclient tests which ship in the distro, see MAPREDUCE-4644.

          So a decision needs to be made whether the -test jars that ship with Hadoop need to be fully functional or not. That's pretty much outside the scope of this JIRA and why I pointed to that other issue.

          Show
          aw Allen Wittenauer added a comment - junit and mockito are currently bundled, and I'm removing them from being bundled in this patch. Also thanks Allen for the pointer jira. I don't think you fully understand what's happening in that other JIRA. Removal of junit was specifically reverted because: This broke the mapreduce jobclient tests which ship in the distro, see MAPREDUCE-4644 . So a decision needs to be made whether the -test jars that ship with Hadoop need to be fully functional or not. That's pretty much outside the scope of this JIRA and why I pointed to that other issue.
          Hide
          xiaochen Xiao Chen added a comment -

          Ah, thanks for pointing out, Allen. Sorry for not reading that jira through and misunderstood.
          Patch 5 keeps the junit and mockito untouched..

          Show
          xiaochen Xiao Chen added a comment - Ah, thanks for pointing out, Allen. Sorry for not reading that jira through and misunderstood. Patch 5 keeps the junit and mockito untouched..
          Hide
          xiaochen Xiao Chen added a comment -

          Hi Tsuyoshi Ozawa,
          Thanks again for helping out. Were you able to try the patch?

          I should have mentioned that the current jdiff scope provided doesn't bundle it into the jars. Making the scope of jdiff to "compile" does make it show up though, so I didn't include that change in the latest patch. Is it okay to have it in our deps, but not bundled? (That is, the as-is option in my above comment)

          Show
          xiaochen Xiao Chen added a comment - Hi Tsuyoshi Ozawa , Thanks again for helping out. Were you able to try the patch? I should have mentioned that the current jdiff scope provided doesn't bundle it into the jars. Making the scope of jdiff to "compile" does make it show up though, so I didn't include that change in the latest patch. Is it okay to have it in our deps, but not bundled? (That is, the as-is option in my above comment )
          Hide
          andrew.wang Andrew Wang added a comment -

          I made the spreadsheet accessible publicly, ping me if you want edit access:

          https://docs.google.com/spreadsheets/d/1HL2b4PSdQMZDVJmum1GIKrteFr2oainApTLiJTPnfd4/edit?usp=sharing

          Reviewing the patch, there are a lot of things listed as "in source distribution" that I didn't think were there, e.g.

          For:
          	  servlet-api 2.5
          	  jsp-api 2.1
          	  JavaBeans Activation Framework
          	  Java Servlet API 3.0.1
          	in source distribution, and
          

          I thought we had all the source distribution items covered already, so these new additions would only apply to the binary distribution.

          There's also a few copies of the GPL and LGPL still in LICENSES. This content was supposed to be pulled from the Licenses tab on the spreadsheet, apparently not?

          Show
          andrew.wang Andrew Wang added a comment - I made the spreadsheet accessible publicly, ping me if you want edit access: https://docs.google.com/spreadsheets/d/1HL2b4PSdQMZDVJmum1GIKrteFr2oainApTLiJTPnfd4/edit?usp=sharing Reviewing the patch, there are a lot of things listed as "in source distribution" that I didn't think were there, e.g. For: servlet-api 2.5 jsp-api 2.1 JavaBeans Activation Framework Java Servlet API 3.0.1 in source distribution, and I thought we had all the source distribution items covered already, so these new additions would only apply to the binary distribution. There's also a few copies of the GPL and LGPL still in LICENSES. This content was supposed to be pulled from the Licenses tab on the spreadsheet, apparently not?
          Hide
          andrew.wang Andrew Wang added a comment -

          Missed the jdiff question:

          jdiff is not bundled. But it is used in source (in hadoop-project/pom.xml). I'm not sure how to proceed with this one... Shall we 1) try remove it completely 2) as-is 3) replace it with something else? We listed jdiff in LICENSE for source distribution (for honesty...)

          If jdiff is not bundled, then we don't need to talk about it in L&N. Mentioning it in a pom.xml file is okay, it'll be downloaded on-demand then. So I vote option 1).

          Thanks again for pushing on this Xiao!

          Show
          andrew.wang Andrew Wang added a comment - Missed the jdiff question: jdiff is not bundled. But it is used in source (in hadoop-project/pom.xml). I'm not sure how to proceed with this one... Shall we 1) try remove it completely 2) as-is 3) replace it with something else? We listed jdiff in LICENSE for source distribution (for honesty...) If jdiff is not bundled, then we don't need to talk about it in L&N. Mentioning it in a pom.xml file is okay, it'll be downloaded on-demand then. So I vote option 1). Thanks again for pushing on this Xiao!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew Wang reviewing.

          I thought we had all the source distribution items covered already, so these new additions would only apply to the binary distribution.

          Maybe I comprehend the L&N wrong. I read this comment from you above and thought we want to list dependencies and say they are in the bundle, or in source code, or both.
          Should we just list anything that has 'bundled?'==Y, and skip others? That will be simple, just wanna make sure before I make the change.

          There's also a few copies of the GPL and LGPL still in LICENSES. This content was supposed to be pulled from the Licenses tab on the spreadsheet, apparently not?

          The script only groups from the dependencies tab, and list the license name currently. After that I went to the links, and copied the license and 80-char-wrapped it. I may have forgotten to remove the GPL part from CDDL+GPL...
          I guess one more automation we can do is to paste that into the License text column and automate.

          So, LGPL will be gone once we get rid of jdiff.
          GPL should be gone since they're all CDDL+GPL w/ CPE. We should be good using CDDL.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew Wang reviewing. I thought we had all the source distribution items covered already, so these new additions would only apply to the binary distribution. Maybe I comprehend the L&N wrong. I read this comment from you above and thought we want to list dependencies and say they are in the bundle, or in source code, or both. Should we just list anything that has 'bundled?'==Y, and skip others? That will be simple, just wanna make sure before I make the change. There's also a few copies of the GPL and LGPL still in LICENSES. This content was supposed to be pulled from the Licenses tab on the spreadsheet, apparently not? The script only groups from the dependencies tab, and list the license name currently. After that I went to the links, and copied the license and 80-char-wrapped it. I may have forgotten to remove the GPL part from CDDL+GPL... I guess one more automation we can do is to paste that into the License text column and automate. So, LGPL will be gone once we get rid of jdiff. GPL should be gone since they're all CDDL+GPL w/ CPE. We should be good using CDDL.
          Hide
          andrew.wang Andrew Wang added a comment -

          AFAIK we have all the source dependencies covered by the existing L&N information. "Source dependencies" means things included in the source tarball, which is still a bundle. It's not exactly the same thing as the source code.

          I wasn't aware of the "bundled" column, I assume it refers to whether something is actually included in the binary tarball? I'd prefer a different name, since both the src and bin tarballs could be considered "bundles". Hopefully whoever added this column (Akira?) can comment; else we can validate by building the release artifacts and filtering based on the contents.

          Show
          andrew.wang Andrew Wang added a comment - AFAIK we have all the source dependencies covered by the existing L&N information. "Source dependencies" means things included in the source tarball, which is still a bundle. It's not exactly the same thing as the source code. I wasn't aware of the "bundled" column, I assume it refers to whether something is actually included in the binary tarball? I'd prefer a different name, since both the src and bin tarballs could be considered "bundles". Hopefully whoever added this column (Akira?) can comment; else we can validate by building the release artifacts and filtering based on the contents.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew, understood, will post a new patch today.
          Yeah the 'bundled' column was added by Akira by looking up all *.jar files in the build output. I'll make my next revision include only those with a 'Y' in the column.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew, understood, will post a new patch today. Yeah the 'bundled' column was added by Akira by looking up all *.jar files in the build output. I'll make my next revision include only those with a 'Y' in the column.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 6 addresses the comments above:

          • Only the 'bundled'==Y deps are included. (I didn't rename the column in the doc, but please feel free to do so)
          • jdiff is hence not included, because it's not bundled.
            (Verified by find ./target/ -name "*jar"|xargs -n1 basename|sort |uniq|grep -v 3.0.0-alpha )

          Let me be thorough to hopefully help review:

          • I put license text into the Licenses tab in the doc, and update the generate script to read that
          • Still had to do the manual 80-char-wrap, because I don't know how to preserve line change after saving that as a tsv, or how to decently break the content with 80-char and preserve paragraphs.
          • The license texts are all copied from the link after it.
          • For the W3C license, the website lists the license and disclaimers separately. Only the 'License' section is included. Please let me know if this is not right.

          If anyone could try the patch or would like to make any changes, please do.

          Show
          xiaochen Xiao Chen added a comment - Patch 6 addresses the comments above: Only the 'bundled'==Y deps are included. (I didn't rename the column in the doc, but please feel free to do so) jdiff is hence not included, because it's not bundled. (Verified by find ./target/ -name "*jar"|xargs -n1 basename|sort |uniq|grep -v 3.0.0-alpha ) Let me be thorough to hopefully help review: I put license text into the Licenses tab in the doc, and update the generate script to read that Still had to do the manual 80-char-wrap, because I don't know how to preserve line change after saving that as a tsv, or how to decently break the content with 80-char and preserve paragraphs. The license texts are all copied from the link after it. For the W3C license, the website lists the license and disclaimers separately. Only the 'License' section is included. Please let me know if this is not right. If anyone could try the patch or would like to make any changes, please do.
          Hide
          andrew.wang Andrew Wang added a comment -

          I made a few improvements to Xiao's patch:

          • Tweaked the pom dependencies so it works with a clean m2 cache
          • Removed "-" from new module name for consistency
          • Changed the LICENSE.txt wording to reflect that these new additions are only for the binary distribution
          • Pulled mockito and slf4j to their own copy of the MIT license, since the existing version for bootstrap has the Twitter copyright

          I wrote the following script to make sure all the generated JARs have the LICENSE and NOTICE:

          for f in $(find . -name "hadoop*SNAPSHOT.jar"); do
              jar -tf $f | grep "LICENSE" > /dev/null
              RET1=$?
              jar -tf $f | grep "NOTICE" > /dev/null
              RET2=$?
          
              if [ $RET1 -ne 0 -a $RET2 -ne 0 ]; then
                  echo $f "missing LICENSE and NOTICE!";
              elif [ $RET1 -ne 0 ]; then
                  echo $f "missing LICENSE!";
              elif [ $RET2 -ne 0 ]; then
                  echo $f "missing NOTICE!";
              fi
          done
          

          This is the output:

          ./hadoop-project-dist/target/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT.jar missing LICENSE and NOTICE!
          ./hadoop-project-dist/target/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT/share/hadoop/UNDEF/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT.jar missing LICENSE and NOTICE!
          ./hadoop-build-tools/target/hadoop-build-tools-3.0.0-alpha1-SNAPSHOT.jar missing LICENSE and NOTICE!

          This is because these inherit from hadoop-main rather than hadoop-project, which is where the remote-resources-plugin is called. There's not much in these JARs, so we might not need to generate/deploy them. If we do, then we can do some special-case L&N work.

          Show
          andrew.wang Andrew Wang added a comment - I made a few improvements to Xiao's patch: Tweaked the pom dependencies so it works with a clean m2 cache Removed "-" from new module name for consistency Changed the LICENSE.txt wording to reflect that these new additions are only for the binary distribution Pulled mockito and slf4j to their own copy of the MIT license, since the existing version for bootstrap has the Twitter copyright I wrote the following script to make sure all the generated JARs have the LICENSE and NOTICE: for f in $(find . -name "hadoop*SNAPSHOT.jar" ); do jar -tf $f | grep "LICENSE" > /dev/ null RET1=$? jar -tf $f | grep "NOTICE" > /dev/ null RET2=$? if [ $RET1 -ne 0 -a $RET2 -ne 0 ]; then echo $f "missing LICENSE and NOTICE!" ; elif [ $RET1 -ne 0 ]; then echo $f "missing LICENSE!" ; elif [ $RET2 -ne 0 ]; then echo $f "missing NOTICE!" ; fi done This is the output: ./hadoop-project-dist/target/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT.jar missing LICENSE and NOTICE! ./hadoop-project-dist/target/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT/share/hadoop/UNDEF/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT.jar missing LICENSE and NOTICE! ./hadoop-build-tools/target/hadoop-build-tools-3.0.0-alpha1-SNAPSHOT.jar missing LICENSE and NOTICE! This is because these inherit from hadoop-main rather than hadoop-project, which is where the remote-resources-plugin is called. There's not much in these JARs, so we might not need to generate/deploy them. If we do, then we can do some special-case L&N work.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks so much Andrew Wang for testing out the patch and putting up a new rev.
          I've applied the new patch 7, and attaching patch 8 which has below improvements:

          • LICENSE.txt and NOTICE.txt under hadoop-resource-bundle doesn't need to be checked in
          • Moved the copy L&N logic from hadoop-resource-bundle/pom.xml to hadoop/pom.xml to make it work
          • IIUC on http://www.apache.org/dev/licensing-howto.html#mod-notice , MIT license doesn't need copyright. So grouped mockito and slf4j back, and removed the original twitter copyright which seems unnecessary.

          Also, the 3 jars that's reported missing actually does not show up in hadoop-dist after maven package and hence not bundled, so IMO they're ok as-is.

          Tested patch 8 builds locally, verified jars under hadoop-dist/target all contain L&N.

          Show
          xiaochen Xiao Chen added a comment - Thanks so much Andrew Wang for testing out the patch and putting up a new rev. I've applied the new patch 7, and attaching patch 8 which has below improvements: LICENSE.txt and NOTICE.txt under hadoop-resource-bundle doesn't need to be checked in Moved the copy L&N logic from hadoop-resource-bundle/pom.xml to hadoop/pom.xml to make it work IIUC on http://www.apache.org/dev/licensing-howto.html#mod-notice , MIT license doesn't need copyright. So grouped mockito and slf4j back, and removed the original twitter copyright which seems unnecessary. Also, the 3 jars that's reported missing actually does not show up in hadoop-dist after maven package and hence not bundled, so IMO they're ok as-is. Tested patch 8 builds locally, verified jars under hadoop-dist/target all contain L&N.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 5s trunk passed
          +1 compile 7m 4s trunk passed
          +1 mvnsite 8m 39s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 javadoc 5m 46s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          -1 mvninstall 0m 8s hadoop-project in the patch failed.
          -1 mvninstall 0m 6s hadoop-project-dist in the patch failed.
          +1 compile 11m 48s the patch passed
          +1 javac 11m 48s the patch passed
          +1 mvnsite 9m 24s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 javadoc 5m 1s the patch passed
          -1 unit 11m 23s root in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          79m 56s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807238/HADOOP-12893.008.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux cfd73ce2bbec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bca31fe
          Default Java 1.8.0_91
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-mvninstall-hadoop-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-unit-root.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/testReport/
          modules C: hadoop-project hadoop-project-dist . hadoop-resource-bundle U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 5s trunk passed +1 compile 7m 4s trunk passed +1 mvnsite 8m 39s trunk passed +1 mvneclipse 0m 46s trunk passed +1 javadoc 5m 46s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch -1 mvninstall 0m 8s hadoop-project in the patch failed. -1 mvninstall 0m 6s hadoop-project-dist in the patch failed. +1 compile 11m 48s the patch passed +1 javac 11m 48s the patch passed +1 mvnsite 9m 24s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 javadoc 5m 1s the patch passed -1 unit 11m 23s root in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 79m 56s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807238/HADOOP-12893.008.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux cfd73ce2bbec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bca31fe Default Java 1.8.0_91 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-mvninstall-hadoop-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/testReport/ modules C: hadoop-project hadoop-project-dist . hadoop-resource-bundle U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Looks like jenkins says trunk compiled fine, but hadoop-project and hadoop-project-dist needs to be compiled as well. I'll try to fix, but not sure what's the best way to handle it.
          If anyone has suggestions or experience with such, feel free to comment or post a patch. Thanks!

          Show
          xiaochen Xiao Chen added a comment - Looks like jenkins says trunk compiled fine, but hadoop-project and hadoop-project-dist needs to be compiled as well. I'll try to fix, but not sure what's the best way to handle it. If anyone has suggestions or experience with such, feel free to comment or post a patch. Thanks!
          Hide
          andrew.wang Andrew Wang added a comment -

          I ran Xiao's latest patch and then my check script, and it seems to have worked. Thanks for fixing my rev!

          If no one objects, I'd like to commit this to trunk tomorrow. More review though would always be appreciated.

          branch-2 and backwards will need some customization, since the deps are different.

          Show
          andrew.wang Andrew Wang added a comment - I ran Xiao's latest patch and then my check script, and it seems to have worked. Thanks for fixing my rev! If no one objects, I'd like to commit this to trunk tomorrow. More review though would always be appreciated. branch-2 and backwards will need some customization, since the deps are different.
          Hide
          aw Allen Wittenauer added a comment - - edited

          I'm guessing the dependencies are in the wrong order, which is why yetus rejected it:

          C: hadoop-project hadoop-project-dist . hadoop-resource-bundle
          

          If hadoop-project and hadoop-project-dist require hadoop-resource-bundle, then that needs to be dealt with before this gets committed.

          EDIT:

          Yup, maven is having problems with that module:

          https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/maven-patch-validate-root.txt

          As a result, Yetus can't figure out the order properly so tacks it onto the end, thus breaking the build when trying to do the dependencies for individual modules.

          FWIW, Yetus' maven dependency bits expects something to generate a line that ends in '@ module-name' so that it can properly order bits.

          Show
          aw Allen Wittenauer added a comment - - edited I'm guessing the dependencies are in the wrong order, which is why yetus rejected it: C: hadoop-project hadoop-project-dist . hadoop-resource-bundle If hadoop-project and hadoop-project-dist require hadoop-resource-bundle, then that needs to be dealt with before this gets committed. EDIT: Yup, maven is having problems with that module: https://builds.apache.org/job/PreCommit-HADOOP-Build/9630/artifact/patchprocess/maven-patch-validate-root.txt As a result, Yetus can't figure out the order properly so tacks it onto the end, thus breaking the build when trying to do the dependencies for individual modules. FWIW, Yetus' maven dependency bits expects something to generate a line that ends in '@ module-name' so that it can properly order bits.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Allen Wittenauer for looking into this. I think these are different problems: the pom in hadoop-resource-bundle didn't have version for its maven-remote-resources-plugin, good catch. I'm attaching patch 9 to fix that.

          I think jenkins is saying hadoop-project cannot be built alone - I can reproduce that locally, but not sure what's the fix yet.

          • remove m2 cache for hadoop-resource-bundle
          • cd hadoop-project (or, any project)
          • maven install, failed.

          Seems we need to tell maven to have hadoop-resource-bundle built before building any of the projects, otherwise it's dependency logic will try to download and fail.

          Show
          xiaochen Xiao Chen added a comment - Thanks Allen Wittenauer for looking into this. I think these are different problems: the pom in hadoop-resource-bundle didn't have version for its maven-remote-resources-plugin , good catch. I'm attaching patch 9 to fix that. I think jenkins is saying hadoop-project cannot be built alone - I can reproduce that locally, but not sure what's the fix yet. remove m2 cache for hadoop-resource-bundle cd hadoop-project (or, any project) maven install, failed. Seems we need to tell maven to have hadoop-resource-bundle built before building any of the projects, otherwise it's dependency logic will try to download and fail.
          Hide
          aw Allen Wittenauer added a comment -

          Put this in the hadoop-resource-bundle's pom.xml:

                <plugin>
                   <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-antrun-plugin</artifactId>
                  <executions>
                    <execution>
                      <id>dummy</id>
                      <phase>validate</phase>
                      <goals>
                        <goal>run</goal>
                      </goals>
                    </execution>
                  </executions>
                </plugin>
          
          Show
          aw Allen Wittenauer added a comment - Put this in the hadoop-resource-bundle's pom.xml: <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-antrun-plugin</artifactId> <executions> <execution> <id>dummy</id> <phase>validate</phase> <goals> <goal>run</goal> </goals> </execution> </executions> </plugin>
          Hide
          aw Allen Wittenauer added a comment -

          Actually, is there some reason this is a separate module from hadoop-build-tools? (It has the same problem, FWIW.)

          Show
          aw Allen Wittenauer added a comment - Actually, is there some reason this is a separate module from hadoop-build-tools? (It has the same problem, FWIW.)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 9m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 47s Maven dependency ordering for branch
          +1 mvninstall 6m 32s trunk passed
          +1 compile 6m 38s trunk passed
          +1 mvnsite 8m 41s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 javadoc 4m 54s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          -1 mvninstall 0m 7s hadoop-project in the patch failed.
          -1 mvninstall 0m 6s hadoop-project-dist in the patch failed.
          +1 compile 6m 49s the patch passed
          +1 javac 6m 49s the patch passed
          +1 mvnsite 8m 37s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 5m 3s the patch passed
          -1 unit 87m 37s root in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          154m 41s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807535/HADOOP-12893.009.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux a3f137e0b439 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 16b1cc7
          Default Java 1.8.0_91
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-mvninstall-hadoop-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-unit-root.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/testReport/
          modules C: hadoop-project hadoop-project-dist . hadoop-resource-bundle U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 9m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 47s Maven dependency ordering for branch +1 mvninstall 6m 32s trunk passed +1 compile 6m 38s trunk passed +1 mvnsite 8m 41s trunk passed +1 mvneclipse 0m 41s trunk passed +1 javadoc 4m 54s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch -1 mvninstall 0m 7s hadoop-project in the patch failed. -1 mvninstall 0m 6s hadoop-project-dist in the patch failed. +1 compile 6m 49s the patch passed +1 javac 6m 49s the patch passed +1 mvnsite 8m 37s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 5m 3s the patch passed -1 unit 87m 37s root in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 154m 41s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807535/HADOOP-12893.009.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux a3f137e0b439 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 16b1cc7 Default Java 1.8.0_91 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-mvninstall-hadoop-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/testReport/ modules C: hadoop-project hadoop-project-dist . hadoop-resource-bundle U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9642/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Allen Wittenauer, I tried the snippet you proposed above, didn't work for me, not sure if I missed anything. Did you try it somehow? (maybe in another project?) I would really appreciate if you could try on the latest patch here.

          I looked more at the hadoop-project build, but seems it's not trying to build hadoop-resource-bundle at all. I tried to add a dependency but didn't work out either... (maven debug output shows REACTOR BUILD PLAN not taking hadoop-resource-bundle...) I wouldn't be able to work more on this until Friday, so if anyone wants to help out and unblock this sooner, please feel free.

          is there some reason this is a separate module from hadoop-build-tools?

          I think having the resource bundle dedicated for L&N, which will be bundled into our package is clearer logically.
          hadoop-build-tools, looking from its name, feels auxiliary, and seems not needed to be bundled into the jars. (only checkstyle stuff)

          Show
          xiaochen Xiao Chen added a comment - Thanks Allen Wittenauer , I tried the snippet you proposed above, didn't work for me, not sure if I missed anything. Did you try it somehow? (maybe in another project?) I would really appreciate if you could try on the latest patch here. I looked more at the hadoop-project build, but seems it's not trying to build hadoop-resource-bundle at all. I tried to add a dependency but didn't work out either... (maven debug output shows REACTOR BUILD PLAN not taking hadoop-resource-bundle...) I wouldn't be able to work more on this until Friday, so if anyone wants to help out and unblock this sooner, please feel free. is there some reason this is a separate module from hadoop-build-tools? I think having the resource bundle dedicated for L&N, which will be bundled into our package is clearer logically. hadoop-build-tools, looking from its name, feels auxiliary, and seems not needed to be bundled into the jars. (only checkstyle stuff)
          Hide
          aw Allen Wittenauer added a comment - - edited

          That change will fix it enough for Apache Yetus to get the dependency ordering correct since mvn validate will print the necessary hints on the screen. (For those playing along and since I'm sure someone else will ask: mvn dependency:* doesn't list plug-ins and since we don't publish the hadoop maven plugin, something else had to be used in Apache Yetus. The alternative is to go back to a manually maintained dependency list in the hadoop personality, but it's been shown that it's extremely hard to get and keep that correct.)

          hadoop-build-tools, looking from its name, feels auxiliary, and seems not needed to be bundled into the jars. (only checkstyle stuff)

          It's literally: stuff used during the build process. Just like L&N....

          Show
          aw Allen Wittenauer added a comment - - edited That change will fix it enough for Apache Yetus to get the dependency ordering correct since mvn validate will print the necessary hints on the screen. (For those playing along and since I'm sure someone else will ask: mvn dependency:* doesn't list plug-ins and since we don't publish the hadoop maven plugin, something else had to be used in Apache Yetus. The alternative is to go back to a manually maintained dependency list in the hadoop personality, but it's been shown that it's extremely hard to get and keep that correct.) hadoop-build-tools, looking from its name, feels auxiliary, and seems not needed to be bundled into the jars. (only checkstyle stuff) It's literally: stuff used during the build process. Just like L&N....
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          branch-2 and backwards will need some customization, since the deps are different.

          Can someone comment on how much effort this is going to be? There are a couple of releases 2.7.3/2.8.0 that have been waiting on this for quite a while.

          Please let me know if you need more help in accelerating this.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - branch-2 and backwards will need some customization, since the deps are different. Can someone comment on how much effort this is going to be? There are a couple of releases 2.7.3/2.8.0 that have been waiting on this for quite a while. Please let me know if you need more help in accelerating this.
          Hide
          andrew.wang Andrew Wang added a comment -

          The spreadsheet tracks some of the branch-2 vs. trunk differences. I'm not sure if we've thoroughly checked all the differences, but verifying and fixing this is maybe half a day of work per branch. Not too bad.

          What we could really use help with is getting the Maven stuff working. If anyone can fix that, it'd be much appreciated.

          Show
          andrew.wang Andrew Wang added a comment - The spreadsheet tracks some of the branch-2 vs. trunk differences. I'm not sure if we've thoroughly checked all the differences, but verifying and fixing this is maybe half a day of work per branch. Not too bad. What we could really use help with is getting the Maven stuff working. If anyone can fix that, it'd be much appreciated.
          Hide
          xiaochen Xiao Chen added a comment - - edited

          Patch 10 addresses what Allen suggested:

          • Use the existing hadoop-build-tools to avoid adding a new project
          • Add the snippet to make maven validate happy
          • Side effect 1: fixes the same problem in build-tools, since now they really become the same problem
          • Side effect 2: I bet my 2 cents on hadoop-project-dist and hadoop-project would pass now.

          Sorry not being able to work on this on Friday, please review and feel free to comment. Thanks again.

          Show
          xiaochen Xiao Chen added a comment - - edited Patch 10 addresses what Allen suggested: Use the existing hadoop-build-tools to avoid adding a new project Add the snippet to make maven validate happy Side effect 1: fixes the same problem in build-tools, since now they really become the same problem Side effect 2: I bet my 2 cents on hadoop-project-dist and hadoop-project would pass now. Sorry not being able to work on this on Friday, please review and feel free to comment. Thanks again.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 7m 2s trunk passed
          +1 compile 7m 24s trunk passed
          +1 mvnsite 10m 38s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 javadoc 6m 19s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 9m 9s the patch passed
          +1 compile 8m 50s the patch passed
          +1 javac 8m 50s the patch passed
          +1 mvnsite 10m 58s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          +1 javadoc 6m 2s the patch passed
          -1 unit 12m 42s root in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          82m 36s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl
            hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808570/HADOOP-12893.10.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 5dc9ceb268ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bddea5f
          Default Java 1.8.0_91
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9675/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9675/testReport/
          modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9675/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 7m 2s trunk passed +1 compile 7m 24s trunk passed +1 mvnsite 10m 38s trunk passed +1 mvneclipse 0m 41s trunk passed +1 javadoc 6m 19s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 9m 9s the patch passed +1 compile 8m 50s the patch passed +1 javac 8m 50s the patch passed +1 mvnsite 10m 58s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 6s The patch has no ill-formed XML file. +1 javadoc 6m 2s the patch passed -1 unit 12m 42s root in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 82m 36s Reason Tests Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808570/HADOOP-12893.10.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 5dc9ceb268ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bddea5f Default Java 1.8.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9675/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9675/testReport/ modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9675/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Xiao Chen for the continuous work.

          • In hadoop-build-tools/pom.xml, we need not to set the version of maven-remote-resources-plugin because it is already set in hadoop-project/pom.xml.
          • In hadoop-project/pom.xml, would you set a parameter as follows and use it to set the version of maven-remote-resources-plugin?
            hadoop-project/pom.xml
                <maven-clean-plugin.version>2.5</maven-clean-plugin.version>
                <maven-compiler-plugin.version>3.1</maven-compiler-plugin.version>
                <maven-install-plugin.version>2.5.1</maven-install-plugin.version>
            
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Xiao Chen for the continuous work. In hadoop-build-tools/pom.xml, we need not to set the version of maven-remote-resources-plugin because it is already set in hadoop-project/pom.xml. In hadoop-project/pom.xml, would you set a parameter as follows and use it to set the version of maven-remote-resources-plugin? hadoop-project/pom.xml <maven-clean-plugin.version>2.5</maven-clean-plugin.version> <maven-compiler-plugin.version>3.1</maven-compiler-plugin.version> <maven-install-plugin.version>2.5.1</maven-install-plugin.version>
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Akira Ajisaka for review and good catch. Patch 011 address both comments. Verified again that all jars under hadoop-dist/target after mvn package from a clean cache contains L&N.

          Show
          xiaochen Xiao Chen added a comment - Thanks Akira Ajisaka for review and good catch. Patch 011 address both comments. Verified again that all jars under hadoop-dist/target after mvn package from a clean cache contains L&N.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 39s trunk passed
          +1 compile 6m 23s trunk passed
          +1 mvnsite 8m 15s trunk passed
          +1 mvneclipse 0m 48s trunk passed
          +1 javadoc 4m 49s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 7m 8s the patch passed
          +1 compile 6m 21s the patch passed
          +1 javac 6m 21s the patch passed
          +1 mvnsite 8m 3s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 javadoc 4m 57s the patch passed
          -1 unit 11m 41s root in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          68m 35s



          Reason Tests
          Failed junit tests hadoop.ipc.TestIPC



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808716/HADOOP-12893.011.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 3ab222a74791 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c14c1b2
          Default Java 1.8.0_91
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9680/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9680/testReport/
          modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9680/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 39s trunk passed +1 compile 6m 23s trunk passed +1 mvnsite 8m 15s trunk passed +1 mvneclipse 0m 48s trunk passed +1 javadoc 4m 49s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 7m 8s the patch passed +1 compile 6m 21s the patch passed +1 javac 6m 21s the patch passed +1 mvnsite 8m 3s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 javadoc 4m 57s the patch passed -1 unit 11m 41s root in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 68m 35s Reason Tests Failed junit tests hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808716/HADOOP-12893.011.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 3ab222a74791 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c14c1b2 Default Java 1.8.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9680/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9680/testReport/ modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9680/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          I have the nittiest of nits, which is that the comment about "hadoop-resource-bundle" should be changed to "hadoop-build-tools", since we folded that in. Otherwise LGTM.

          I'll wait until business hours PST tomorrow to give Akira a chance to review again. The comment fix seems small enough that we can fix at commit time.

          Show
          andrew.wang Andrew Wang added a comment - I have the nittiest of nits, which is that the comment about "hadoop-resource-bundle" should be changed to "hadoop-build-tools", since we folded that in. Otherwise LGTM. I'll wait until business hours PST tomorrow to give Akira a chance to review again. The comment fix seems small enough that we can fix at commit time.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, thanks Xiao and Andrew.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, thanks Xiao and Andrew.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the great work here everyone, particularly Xiao and Akira. I've committed the 011 patch (with my comment fixup) to trunk.

          Hoping some others can help with the branch-2 backports. I described what's involved in an earlier comment, but it's basically diffing the list of dependencies with trunk, then updating versions and L&N as necessary.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the great work here everyone, particularly Xiao and Akira. I've committed the 011 patch (with my comment fixup) to trunk. Hoping some others can help with the branch-2 backports. I described what's involved in an earlier comment, but it's basically diffing the list of dependencies with trunk, then updating versions and L&N as necessary.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9940 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9940/)
          HADOOP-12893. Verify LICENSE.txt and NOTICE.txt. Contributed by Xiao (wang: rev e383b732c54c542482b0b836e2d2c46eb49b4e2d)

          • hadoop-build-tools/pom.xml
          • LICENSE.txt
          • hadoop-project/pom.xml
          • pom.xml
          • hadoop-project-dist/pom.xml
          • NOTICE.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9940 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9940/ ) HADOOP-12893 . Verify LICENSE.txt and NOTICE.txt. Contributed by Xiao (wang: rev e383b732c54c542482b0b836e2d2c46eb49b4e2d) hadoop-build-tools/pom.xml LICENSE.txt hadoop-project/pom.xml pom.xml hadoop-project-dist/pom.xml NOTICE.txt
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks everyone for the work together! So it probably proves I can't be a lawyer, but I enjoyed working with you all.

          It seems the way we build L&N created some precommit spam. I'm trying to fix that via HADOOP-13258.

          Show
          xiaochen Xiao Chen added a comment - Thanks everyone for the work together! So it probably proves I can't be a lawyer, but I enjoyed working with you all. It seems the way we build L&N created some precommit spam. I'm trying to fix that via HADOOP-13258 .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rebased for branch-2.

          • Removed trailing whitespaces in CHANGES.txt.
          • Removed re2j, which is not in branch-2.
          Show
          ajisakaa Akira Ajisaka added a comment - Rebased for branch-2. Removed trailing whitespaces in CHANGES.txt. Removed re2j, which is not in branch-2.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The branch-2 patch applies to branch-2.8 cleanly.

          Show
          ajisakaa Akira Ajisaka added a comment - The branch-2 patch applies to branch-2.8 cleanly.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch for branch-2.7.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch for branch-2.7.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 44s Maven dependency ordering for branch
          +1 mvninstall 6m 38s branch-2 passed
          +1 compile 5m 44s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 19s branch-2 passed with JDK v1.7.0_101
          +1 mvnsite 10m 17s branch-2 passed
          +1 mvneclipse 0m 49s branch-2 passed
          +1 javadoc 5m 49s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 6m 36s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 7m 29s the patch passed
          +1 compile 7m 40s the patch passed with JDK v1.8.0_91
          +1 javac 7m 40s the patch passed
          +1 compile 7m 4s the patch passed with JDK v1.7.0_101
          +1 javac 7m 4s the patch passed
          +1 mvnsite 10m 3s the patch passed
          +1 mvneclipse 1m 0s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 5m 40s the patch passed with JDK v1.8.0_91
          +1 javadoc 6m 38s the patch passed with JDK v1.7.0_101
          -1 unit 13m 30s root in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          108m 37s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.security.authentication.util.TestZKSignerSecretProvider
          JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809749/HADOOP-12893.branch-2.01.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux c0ceee798892 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 63da641
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/artifact/patchprocess/patch-unit-root-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/testReport/
          modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 44s Maven dependency ordering for branch +1 mvninstall 6m 38s branch-2 passed +1 compile 5m 44s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 19s branch-2 passed with JDK v1.7.0_101 +1 mvnsite 10m 17s branch-2 passed +1 mvneclipse 0m 49s branch-2 passed +1 javadoc 5m 49s branch-2 passed with JDK v1.8.0_91 +1 javadoc 6m 36s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 7m 29s the patch passed +1 compile 7m 40s the patch passed with JDK v1.8.0_91 +1 javac 7m 40s the patch passed +1 compile 7m 4s the patch passed with JDK v1.7.0_101 +1 javac 7m 4s the patch passed +1 mvnsite 10m 3s the patch passed +1 mvneclipse 1m 0s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 5m 40s the patch passed with JDK v1.8.0_91 +1 javadoc 6m 38s the patch passed with JDK v1.7.0_101 -1 unit 13m 30s root in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 108m 37s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.security.authentication.util.TestZKSignerSecretProvider JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809749/HADOOP-12893.branch-2.01.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux c0ceee798892 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 63da641 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/artifact/patchprocess/patch-unit-root-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/testReport/ modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9754/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 6m 34s branch-2.7 passed
          +1 compile 6m 39s branch-2.7 passed with JDK v1.8.0_91
          +1 compile 6m 49s branch-2.7 passed with JDK v1.7.0_101
          +1 mvnsite 9m 3s branch-2.7 passed
          +1 mvneclipse 0m 44s branch-2.7 passed
          +1 javadoc 6m 45s branch-2.7 passed with JDK v1.8.0_91
          +1 javadoc 8m 51s branch-2.7 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          -1 mvninstall 0m 8s hadoop-project in the patch failed.
          -1 mvninstall 0m 6s hadoop-project-dist in the patch failed.
          +1 compile 6m 53s the patch passed with JDK v1.8.0_91
          +1 javac 6m 53s the patch passed
          +1 compile 7m 2s the patch passed with JDK v1.7.0_101
          +1 javac 7m 2s the patch passed
          +1 mvnsite 8m 43s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          -1 whitespace 0m 0s The patch has 6953 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 2m 52s The patch 180 line(s) with tabs.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 6m 30s the patch passed with JDK v1.8.0_91
          +1 javadoc 8m 9s the patch passed with JDK v1.7.0_101
          -1 unit 26m 41s root in the patch failed with JDK v1.7.0_101.
          -1 asflicense 0m 37s The patch generated 1 ASF License warnings.
          148m 14s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.conf.TestConfiguration
          JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809752/HADOOP-12893.branch-2.7.01.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 1c457681a594 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 5a3fed0
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-mvninstall-hadoop-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-unit-root-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-project-dist hadoop-build-tools . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 6m 34s branch-2.7 passed +1 compile 6m 39s branch-2.7 passed with JDK v1.8.0_91 +1 compile 6m 49s branch-2.7 passed with JDK v1.7.0_101 +1 mvnsite 9m 3s branch-2.7 passed +1 mvneclipse 0m 44s branch-2.7 passed +1 javadoc 6m 45s branch-2.7 passed with JDK v1.8.0_91 +1 javadoc 8m 51s branch-2.7 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch -1 mvninstall 0m 8s hadoop-project in the patch failed. -1 mvninstall 0m 6s hadoop-project-dist in the patch failed. +1 compile 6m 53s the patch passed with JDK v1.8.0_91 +1 javac 6m 53s the patch passed +1 compile 7m 2s the patch passed with JDK v1.7.0_101 +1 javac 7m 2s the patch passed +1 mvnsite 8m 43s the patch passed +1 mvneclipse 0m 43s the patch passed -1 whitespace 0m 0s The patch has 6953 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 2m 52s The patch 180 line(s) with tabs. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 6m 30s the patch passed with JDK v1.8.0_91 +1 javadoc 8m 9s the patch passed with JDK v1.7.0_101 -1 unit 26m 41s root in the patch failed with JDK v1.7.0_101. -1 asflicense 0m 37s The patch generated 1 ASF License warnings. 148m 14s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.conf.TestConfiguration JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809752/HADOOP-12893.branch-2.7.01.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 1c457681a594 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 5a3fed0 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-mvninstall-hadoop-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-unit-root-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-project-dist hadoop-build-tools . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9755/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reverted this to close HADOOP-13258. I'll attach a patch without trailing whitespaces shortly.

          Show
          ajisakaa Akira Ajisaka added a comment - Reverted this to close HADOOP-13258 . I'll attach a patch without trailing whitespaces shortly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 8m 17s trunk passed
          +1 compile 8m 28s trunk passed
          +1 mvnsite 10m 33s trunk passed
          +1 mvneclipse 0m 48s trunk passed
          +1 javadoc 5m 59s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 9m 55s the patch passed
          +1 compile 7m 44s the patch passed
          +1 javac 7m 44s the patch passed
          +1 mvnsite 9m 27s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 javadoc 4m 52s the patch passed
          -1 unit 11m 22s root in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          80m 32s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810261/HADOOP-12893.012.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 60e027f886ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ee55b74
          Default Java 1.8.0_91
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9765/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9765/testReport/
          modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9765/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 8m 17s trunk passed +1 compile 8m 28s trunk passed +1 mvnsite 10m 33s trunk passed +1 mvneclipse 0m 48s trunk passed +1 javadoc 5m 59s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 9m 55s the patch passed +1 compile 7m 44s the patch passed +1 javac 7m 44s the patch passed +1 mvnsite 9m 27s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 javadoc 4m 52s the patch passed -1 unit 11m 22s root in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 80m 32s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810261/HADOOP-12893.012.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 60e027f886ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ee55b74 Default Java 1.8.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9765/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9765/testReport/ modules C: hadoop-build-tools hadoop-project hadoop-project-dist . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9765/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Akira for the new patch, LGTM. Also verified the new patch and branch-2 patch work, jars contain L&N.

          Show
          xiaochen Xiao Chen added a comment - Thanks Akira for the new patch, LGTM. Also verified the new patch and branch-2 patch work, jars contain L&N.
          Hide
          andrew.wang Andrew Wang added a comment -

          +1

          Show
          andrew.wang Andrew Wang added a comment - +1
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks for works from Akira Ajisaka, Andrew Wang, Xiao Chen.

          Tried to apply branch-2 patch on branch-2.8, verified artifacts contains new changes. +1 to latest patch.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks for works from Akira Ajisaka , Andrew Wang , Xiao Chen . Tried to apply branch-2 patch on branch-2.8, verified artifacts contains new changes. +1 to latest patch.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Tx for taking this to completion, Akira Ajisaka, Xiao Chen and Andrew Wang.

          There are enough +1s here, if someone can push this into branch-2, branch-2.8 and branch-2.7.3, I can finally move forward with 2.7.3 (first and later 2.8) after waiting on this for 3 months. If not, I'll myself push this in tomorrow.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Tx for taking this to completion, Akira Ajisaka , Xiao Chen and Andrew Wang . There are enough +1s here, if someone can push this into branch-2, branch-2.8 and branch-2.7.3, I can finally move forward with 2.7.3 (first and later 2.8) after waiting on this for 3 months. If not, I'll myself push this in tomorrow.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Xiao Chen, Andrew Wang, and Wangda Tan for review. Committed to trunk, branch-2, and branch-2.8.
          I need to modify the branch-2.7 patch a little because the project.version has changed. I'll provide patches for branch-2.7 and branch-2.7.3 shortly.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Xiao Chen , Andrew Wang , and Wangda Tan for review. Committed to trunk, branch-2, and branch-2.8. I need to modify the branch-2.7 patch a little because the project.version has changed. I'll provide patches for branch-2.7 and branch-2.7.3 shortly.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Modified project version in hadoop-build-tools/pom.xml

          • branch-2.7: 2.7.3-SNAPSHOT -> 2.7.4-SNAPSHOT
          • branch-2.7.3: 2.7.3-SNAPSHOT -> 2.7.3
          Show
          ajisakaa Akira Ajisaka added a comment - Modified project version in hadoop-build-tools/pom.xml branch-2.7: 2.7.3-SNAPSHOT -> 2.7.4-SNAPSHOT branch-2.7.3: 2.7.3-SNAPSHOT -> 2.7.3
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The above change is very small, so I'll commit to branch-2.7 and branch-2.7.3 tomorrow JST if there are no objections.

          Show
          ajisakaa Akira Ajisaka added a comment - The above change is very small, so I'll commit to branch-2.7 and branch-2.7.3 tomorrow JST if there are no objections.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Xiao Chen, I have a question about this issue.

          NOTICE.txt
          Also, please refer to each LICENSE.<component>.txt file, which is located in
          the 'license' directory of the distribution file, for the license terms of the
          components that this product depends on.
          

          LICENSE.<component>.txt files are not included in the patch. Should we add them in a separate jira?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Xiao Chen , I have a question about this issue. NOTICE.txt Also, please refer to each LICENSE.<component>.txt file, which is located in the 'license' directory of the distribution file, for the license terms of the components that this product depends on. LICENSE.<component>.txt files are not included in the patch. Should we add them in a separate jira?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 7m 32s branch-2.7.3 passed
          +1 compile 5m 12s branch-2.7.3 passed with JDK v1.8.0_91
          +1 compile 6m 37s branch-2.7.3 passed with JDK v1.7.0_101
          +1 mvnsite 8m 42s branch-2.7.3 passed
          +1 mvneclipse 1m 10s branch-2.7.3 passed
          +1 javadoc 5m 2s branch-2.7.3 passed with JDK v1.8.0_91
          +1 javadoc 8m 13s branch-2.7.3 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          -1 mvninstall 0m 9s hadoop-project in the patch failed.
          -1 mvninstall 0m 8s hadoop-project-dist in the patch failed.
          +1 compile 5m 40s the patch passed with JDK v1.8.0_91
          +1 javac 5m 40s the patch passed
          +1 compile 6m 24s the patch passed with JDK v1.7.0_101
          +1 javac 6m 24s the patch passed
          +1 mvnsite 8m 10s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          -1 whitespace 0m 0s The patch has 7753 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 3m 48s The patch 184 line(s) with tabs.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 4m 52s the patch passed with JDK v1.8.0_91
          +1 javadoc 8m 10s the patch passed with JDK v1.7.0_101
          -1 unit 34m 37s root in the patch failed with JDK v1.7.0_101.
          -1 asflicense 1m 0s The patch generated 1 ASF License warnings.
          172m 46s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.ipc.TestDecayRpcScheduler
          JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
            org.apache.hadoop.conf.TestConfiguration
          JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810766/HADOOP-12893.branch-2.7.3.01.patch
          JIRA Issue HADOOP-12893
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 9093298e00fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7.3 / ff327bc
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-mvninstall-hadoop-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-unit-root-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-project-dist hadoop-build-tools . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 7m 32s branch-2.7.3 passed +1 compile 5m 12s branch-2.7.3 passed with JDK v1.8.0_91 +1 compile 6m 37s branch-2.7.3 passed with JDK v1.7.0_101 +1 mvnsite 8m 42s branch-2.7.3 passed +1 mvneclipse 1m 10s branch-2.7.3 passed +1 javadoc 5m 2s branch-2.7.3 passed with JDK v1.8.0_91 +1 javadoc 8m 13s branch-2.7.3 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch -1 mvninstall 0m 9s hadoop-project in the patch failed. -1 mvninstall 0m 8s hadoop-project-dist in the patch failed. +1 compile 5m 40s the patch passed with JDK v1.8.0_91 +1 javac 5m 40s the patch passed +1 compile 6m 24s the patch passed with JDK v1.7.0_101 +1 javac 6m 24s the patch passed +1 mvnsite 8m 10s the patch passed +1 mvneclipse 0m 44s the patch passed -1 whitespace 0m 0s The patch has 7753 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 3m 48s The patch 184 line(s) with tabs. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 4m 52s the patch passed with JDK v1.8.0_91 +1 javadoc 8m 10s the patch passed with JDK v1.7.0_101 -1 unit 34m 37s root in the patch failed with JDK v1.7.0_101. -1 asflicense 1m 0s The patch generated 1 ASF License warnings. 172m 46s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.ipc.TestDecayRpcScheduler JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken   org.apache.hadoop.conf.TestConfiguration JDK v1.7.0_101 Timed out junit tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810766/HADOOP-12893.branch-2.7.3.01.patch JIRA Issue HADOOP-12893 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 9093298e00fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7.3 / ff327bc Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-mvninstall-hadoop-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-mvninstall-hadoop-project-dist.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-unit-root-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-project-dist hadoop-build-tools . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9780/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Hi Akira Ajisaka,
          I believe that is from within the netty notice included in Gson. It's referring to the files inside the netty jar:

          META-INF/license/LICENSE.base64.txt
          META-INF/license/LICENSE.commons-logging.txt
          META-INF/license/LICENSE.felix.txt
          META-INF/license/LICENSE.jboss-logging.txt
          META-INF/license/LICENSE.jsr166y.txt
          META-INF/license/LICENSE.jzlib.txt
          META-INF/license/LICENSE.log4j.txt
          META-INF/license/LICENSE.protobuf.txt
          META-INF/license/LICENSE.slf4j.txt
          META-INF/license/LICENSE.webbit.txt
          

          So IMHO we can leave it as-is.

          Show
          xiaochen Xiao Chen added a comment - Hi Akira Ajisaka , I believe that is from within the netty notice included in Gson. It's referring to the files inside the netty jar: META-INF/license/LICENSE.base64.txt META-INF/license/LICENSE.commons-logging.txt META-INF/license/LICENSE.felix.txt META-INF/license/LICENSE.jboss-logging.txt META-INF/license/LICENSE.jsr166y.txt META-INF/license/LICENSE.jzlib.txt META-INF/license/LICENSE.log4j.txt META-INF/license/LICENSE.protobuf.txt META-INF/license/LICENSE.slf4j.txt META-INF/license/LICENSE.webbit.txt So IMHO we can leave it as-is.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I got it. Thanks a lot!

          Show
          ajisakaa Akira Ajisaka added a comment - I got it. Thanks a lot!
          Hide
          aw Allen Wittenauer added a comment - - edited

          META-INF/license/LICENSE.jboss-logging.txt

          Oh, this is problematic. JBoss is LGPL 2.1. Strictly forbidden.

          Show
          aw Allen Wittenauer added a comment - - edited META-INF/license/LICENSE.jboss-logging.txt Oh, this is problematic. JBoss is LGPL 2.1. Strictly forbidden.
          Hide
          aw Allen Wittenauer added a comment -

          From my quick pass over the netty source, it looks like they include the license files of their optional components in their jar. So, not just their actual/bundled dependencies. Which is, frankly, weird.

          Show
          aw Allen Wittenauer added a comment - From my quick pass over the netty source, it looks like they include the license files of their optional components in their jar. So, not just their actual/bundled dependencies. Which is, frankly, weird.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed to branch-2.7 and branch-2.7.3. Thanks to all who contributed to this issue.

          Which is, frankly, weird.

          Agreed. If we need to fix it, please file a separate jira.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed to branch-2.7 and branch-2.7.3. Thanks to all who contributed to this issue. Which is, frankly, weird. Agreed. If we need to fix it, please file a separate jira.
          Hide
          ebadger Eric Badger added a comment -

          Akira Ajisaka, Allen Wittenauer, Xiao Chen, branch-2.7 is failing to build for me after this patch was committed.

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hadoop-project: Resources archive cannot be found. Failure to find org.apache.hadoop:hadoop-build-tools:jar:2.7.4-SNAPSHOT in http://weakdinner:8081/nexus/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of nexus has elapsed or updates are forced
          

          If I move the hadoop-build-tools module ahead of hadoop-project in the hadoop/pom.xml file, then it builds fine. So, it looks as if there's a missed dependency in hadoop-project and hadoop-build-tools isn't available when it's required. The build also succeeds if you mvn install in the hadoop-build-tools directory before doing a build from the top level of hadoop.

          107   <modules>
          108     <module>hadoop-project</module>
          109     <module>hadoop-project-dist</module>
          110     <module>hadoop-assemblies</module>
          111     <module>hadoop-maven-plugins</module>
          112     <module>hadoop-common-project</module>
          113     <module>hadoop-hdfs-project</module>
          114     <module>hadoop-yarn-project</module>
          115     <module>hadoop-mapreduce-project</module>
          116     <module>hadoop-tools</module>
          117     <module>hadoop-dist</module>
          118     <module>hadoop-client</module>
          119     <module>hadoop-minicluster</module>
          120     <module>hadoop-build-tools</module>
          121   </modules>
          
          Show
          ebadger Eric Badger added a comment - Akira Ajisaka , Allen Wittenauer , Xiao Chen , branch-2.7 is failing to build for me after this patch was committed. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hadoop-project: Resources archive cannot be found. Failure to find org.apache.hadoop:hadoop-build-tools:jar:2.7.4-SNAPSHOT in http://weakdinner:8081/nexus/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of nexus has elapsed or updates are forced If I move the hadoop-build-tools module ahead of hadoop-project in the hadoop/pom.xml file, then it builds fine. So, it looks as if there's a missed dependency in hadoop-project and hadoop-build-tools isn't available when it's required. The build also succeeds if you mvn install in the hadoop-build-tools directory before doing a build from the top level of hadoop. 107 <modules> 108 <module>hadoop-project</module> 109 <module>hadoop-project-dist</module> 110 <module>hadoop-assemblies</module> 111 <module>hadoop-maven-plugins</module> 112 <module>hadoop-common-project</module> 113 <module>hadoop-hdfs-project</module> 114 <module>hadoop-yarn-project</module> 115 <module>hadoop-mapreduce-project</module> 116 <module>hadoop-tools</module> 117 <module>hadoop-dist</module> 118 <module>hadoop-client</module> 119 <module>hadoop-minicluster</module> 120 <module>hadoop-build-tools</module> 121 </modules>
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Eric Badger for reporting this. This error happens in branch-2.6 as well.

          If I move the hadoop-build-tools module ahead of hadoop-project in the hadoop/pom.xml file, then it builds fine.

          I'm +1 for doing this. May I revert this commit from branch-2.7/2.7.3/2.6 and provide patches?
          cc: Andrew Wang, Wangda Tan

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Eric Badger for reporting this. This error happens in branch-2.6 as well. If I move the hadoop-build-tools module ahead of hadoop-project in the hadoop/pom.xml file, then it builds fine. I'm +1 for doing this. May I revert this commit from branch-2.7/2.7.3/2.6 and provide patches? cc: Andrew Wang , Wangda Tan
          Hide
          busbey Sean Busbey added a comment -

          we should instead list hadoop-build-tools as a dependency of hadoop-project so that maven will correctly order the modules. Relying on pom module definition order is brittle and coincidental behavior.

          Show
          busbey Sean Busbey added a comment - we should instead list hadoop-build-tools as a dependency of hadoop-project so that maven will correctly order the modules. Relying on pom module definition order is brittle and coincidental behavior.
          Hide
          ebadger Eric Badger added a comment - - edited

          If hadoop-project depends on hadoop-build-tools, shouldn't we make an explicit dependency on hadoop-build-tools rather than rely on specifying a specific module build order? I tried adding hadoop-build-tools as a dependency in the hadoop-project pom.xml file, but was unable to get the build to succeed. However, that seems to me to be the better fix. I'm not a maven expert, so input from someone with more expertise in this area would be good.

          Edit: I should've reloaded the page before adding my comment. Suffice it to say that I agree with Sean Busbey.

          Show
          ebadger Eric Badger added a comment - - edited If hadoop-project depends on hadoop-build-tools, shouldn't we make an explicit dependency on hadoop-build-tools rather than rely on specifying a specific module build order? I tried adding hadoop-build-tools as a dependency in the hadoop-project pom.xml file, but was unable to get the build to succeed. However, that seems to me to be the better fix. I'm not a maven expert, so input from someone with more expertise in this area would be good. Edit: I should've reloaded the page before adding my comment. Suffice it to say that I agree with Sean Busbey .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Sean Busbey and Eric Badger for the comments.
          Attaching a patch to add hadoop-build-tools as a dependency of hadoop-project. Is it correct?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Sean Busbey and Eric Badger for the comments. Attaching a patch to add hadoop-build-tools as a dependency of hadoop-project. Is it correct?
          Hide
          ebadger Eric Badger added a comment -

          I'm not sure if this is the correct fix, but I have confirmed that this patch does fix the build for branch-2.7.

          Show
          ebadger Eric Badger added a comment - I'm not sure if this is the correct fix, but I have confirmed that this patch does fix the build for branch-2.7.
          Hide
          busbey Sean Busbey added a comment - - edited

          -1 (non-binding) on the addendum. the current patch moves the module definition from its current home to be a child module of the hadoop-project, without changing its location in the filesystem. it would be better to instead just declared hadoop-build-tools as a dependency of hadoop-project (in the dependencies section of the pom)

          Show
          busbey Sean Busbey added a comment - - edited -1 (non-binding) on the addendum. the current patch moves the module definition from its current home to be a child module of the hadoop-project, without changing its location in the filesystem. it would be better to instead just declared hadoop-build-tools as a dependency of hadoop-project (in the dependencies section of the pom)
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thank you for the comments.

          the current patch moves the module definition from its current home to be a child module of the hadoop-project, without changing its location in the filesystem.

          Agreed, but as Eric said, adding hadoop-build-tools as a dependency does not fix the build. (I can reproduce the failure in branch-2.7.3) I'm thinking we need to move the module definition to fix the build. Should we move the module definition and change the location in the filesystem?

          FYI: Now we can build trunk, branch-2, branch-2.8, and branch-2.7 without any fix because hadoop-build-tools jar and pom are uploaded to snapshot repository.
          https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/

          Show
          ajisakaa Akira Ajisaka added a comment - Thank you for the comments. the current patch moves the module definition from its current home to be a child module of the hadoop-project, without changing its location in the filesystem. Agreed, but as Eric said, adding hadoop-build-tools as a dependency does not fix the build. (I can reproduce the failure in branch-2.7.3) I'm thinking we need to move the module definition to fix the build. Should we move the module definition and change the location in the filesystem? FYI: Now we can build trunk, branch-2, branch-2.8, and branch-2.7 without any fix because hadoop-build-tools jar and pom are uploaded to snapshot repository. https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/
          Hide
          busbey Sean Busbey added a comment -

          the current patch moves the module definition from its current home to be a child module of the hadoop-project, without changing its location in the filesystem.

          Agreed, but as Eric said, adding hadoop-build-tools as a dependency does not fix the build. (I can reproduce the failure in branch-2.7.3) I'm thinking we need to move the module definition to fix the build. Should we move the module definition and change the location in the filesystem?

          Since the immediate problem is masked at the moment, could you make a separate jira to track this dependency graph error and I'll look into fixing it?

          Show
          busbey Sean Busbey added a comment - the current patch moves the module definition from its current home to be a child module of the hadoop-project, without changing its location in the filesystem. Agreed, but as Eric said, adding hadoop-build-tools as a dependency does not fix the build. (I can reproduce the failure in branch-2.7.3) I'm thinking we need to move the module definition to fix the build. Should we move the module definition and change the location in the filesystem? Since the immediate problem is masked at the moment, could you make a separate jira to track this dependency graph error and I'll look into fixing it?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Sure! Filed HADOOP-13297.

          Show
          ajisakaa Akira Ajisaka added a comment - Sure! Filed HADOOP-13297 .
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi all, thanks for taking on this monumental task.

          Do you think we need to include a notice for the jcip-annotation bundled jars per their README.

          Copyright (c) 2005 Brian Goetz and Tim Peierls
          Released under the Creative Commons Attribution License
          (http://creativecommons.org/licenses/by/2.5)
          Official home: http://www.jcip.net
          
          Show
          arpitagarwal Arpit Agarwal added a comment - Hi all, thanks for taking on this monumental task. Do you think we need to include a notice for the jcip-annotation bundled jars per their README . Copyright (c) 2005 Brian Goetz and Tim Peierls Released under the Creative Commons Attribution License (http: //creativecommons.org/licenses/by/2.5) Official home: http: //www.jcip.net
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Arpit Agarwal for looking into this.

          Per ASF requirement, NOTICE file is intended for explicit NOTICE by the dependencies, not listing copyright info. (I had similar misconception too when working on the spreadsheet). So for jcip, we're correct to have it's NOTICE to be empty.

          If you see anything else missing, please feel free to comment.

          Show
          xiaochen Xiao Chen added a comment - Thanks Arpit Agarwal for looking into this. Per ASF requirement , NOTICE file is intended for explicit NOTICE by the dependencies, not listing copyright info. (I had similar misconception too when working on the spreadsheet). So for jcip, we're correct to have it's NOTICE to be empty. If you see anything else missing, please feel free to comment.
          Hide
          vinayrpet Vinayakumar B added a comment -

          FYI: Now we can build trunk, branch-2, branch-2.8, and branch-2.7 without any fix because hadoop-build-tools jar and pom are uploaded to snapshot repository.

          I had pushed branch-2.7 artifacts to unblock the 2.7 QA run on 17th June.

          Show
          vinayrpet Vinayakumar B added a comment - FYI: Now we can build trunk, branch-2, branch-2.8, and branch-2.7 without any fix because hadoop-build-tools jar and pom are uploaded to snapshot repository. I had pushed branch-2.7 artifacts to unblock the 2.7 QA run on 17th June.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks all for taking a look at this issue.

          Xiao Chen I'm very sorry for the delay.

          I should have mentioned that the current jdiff scope provided doesn't bundle it into the jars. Making the scope of jdiff to "compile" does make it show up though, so I didn't include that change in the latest patch. Is it okay to have it in our deps, but not bundled? (That is, the as-is option in my above comment)

          You're right. Thank you for fixing it.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks all for taking a look at this issue. Xiao Chen I'm very sorry for the delay. I should have mentioned that the current jdiff scope provided doesn't bundle it into the jars. Making the scope of jdiff to "compile" does make it show up though, so I didn't include that change in the latest patch. Is it okay to have it in our deps, but not bundled? (That is, the as-is option in my above comment) You're right. Thank you for fixing it.
          Hide
          busbey Sean Busbey added a comment -

          Per ASF requirement, NOTICE file is intended for explicit NOTICE by the dependencies, not listing copyright info. (I had similar misconception too when working on the spreadsheet). So for jcip, we're correct to have it's NOTICE to be empty.

          That is specifically talking about the copyright notifications from Category A licenses (like MIT and BSD-3). CC-BY is a Category B license and should get called out ref cat b. The call out can be in either README or NOTICE, but so far it looks like we're taking the NOTICE route. (it also needs to be in the LICENSE file for any artifacts that bundle it, like all non-ASLv2 bundled third party works)

          Show
          busbey Sean Busbey added a comment - Per ASF requirement, NOTICE file is intended for explicit NOTICE by the dependencies, not listing copyright info. (I had similar misconception too when working on the spreadsheet). So for jcip, we're correct to have it's NOTICE to be empty. That is specifically talking about the copyright notifications from Category A licenses (like MIT and BSD-3). CC-BY is a Category B license and should get called out ref cat b . The call out can be in either README or NOTICE, but so far it looks like we're taking the NOTICE route. (it also needs to be in the LICENSE file for any artifacts that bundle it, like all non-ASLv2 bundled third party works)
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the explanation Xiao Chen. For jcip, their licensing terms appear to require we include the copyright and license notice. I can make this update if we agree it is necessary.

          Any republication or derived work distributed in source code form
          must include this copyright and license notice.
          

          Also I think we need to include references to bundled permissively-licensed works in LICENSE.txt as Sean described above.

          That's correct, you should not include anything from BSD or MIT licensed deps in NOTICE. That includes bundled JARs; in both source and bundled binary cases you should have a reference in the LICENSE file for the included work. (ref licensing howto on permissive licenses)

          http://www.apache.org/dev/licensing-howto.html#permissive-deps

          e.g. we bundle okio but there is no reference to it in NOTICE.txt. I still haven't done a full audit so there may be more. I can help fix this too.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the explanation Xiao Chen . For jcip, their licensing terms appear to require we include the copyright and license notice. I can make this update if we agree it is necessary. Any republication or derived work distributed in source code form must include this copyright and license notice. Also I think we need to include references to bundled permissively-licensed works in LICENSE.txt as Sean described above. That's correct, you should not include anything from BSD or MIT licensed deps in NOTICE. That includes bundled JARs; in both source and bundled binary cases you should have a reference in the LICENSE file for the included work. (ref licensing howto on permissive licenses) http://www.apache.org/dev/licensing-howto.html#permissive-deps e.g. we bundle okio but there is no reference to it in NOTICE.txt. I still haven't done a full audit so there may be more. I can help fix this too.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the comments all, and Sean for the explanation. Looks like we need more changes for category B then.

          Arpit Agarwal, could you edit the spreadsheet, and generate + merge the new L&N? That way we don't lose any changes during the iterations. I can also do a pass this week after you're done. Thank you.

          e.g. we bundle okio but there is no reference to it in NOTICE.txt. I still haven't done a full audit so there may be more. I can help fix this too

          This is ASFv2, and I don't see any NOTICE on https://github.com/square/okio ?

          Show
          xiaochen Xiao Chen added a comment - Thanks for the comments all, and Sean for the explanation. Looks like we need more changes for category B then. Arpit Agarwal , could you edit the spreadsheet , and generate + merge the new L&N? That way we don't lose any changes during the iterations. I can also do a pass this week after you're done. Thank you. e.g. we bundle okio but there is no reference to it in NOTICE.txt. I still haven't done a full audit so there may be more. I can help fix this too This is ASFv2, and I don't see any NOTICE on https://github.com/square/okio ?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          This is ASFv2, and I don't see any NOTICE on https://github.com/square/okio ?

          Sorry I meant to say LICENSE.txt. Here is what the guidelines say:

          In LICENSE, add a pointer to the dependency's license within the distribution and a short note summarizing its licensing:

          So it sounds like we need a pointer in LICENSE.txt, for okio that might be something like:

          This product bundles Okio 1.4.0, which is available under a
          Apache license.  For details, see https://github.com/square/okio/blob/master/LICENSE.txt.
          
          Show
          arpitagarwal Arpit Agarwal added a comment - This is ASFv2, and I don't see any NOTICE on https://github.com/square/okio ? Sorry I meant to say LICENSE.txt. Here is what the guidelines say: In LICENSE, add a pointer to the dependency's license within the distribution and a short note summarizing its licensing: So it sounds like we need a pointer in LICENSE.txt, for okio that might be something like: This product bundles Okio 1.4.0, which is available under a Apache license. For details, see https: //github.com/square/okio/blob/master/LICENSE.txt.
          Hide
          busbey Sean Busbey added a comment -

          LICENSE already contains the ASLv2 license covering the aggregate work, including Okio. You need not list the individual components of an aggregate work made up of ASLv2 licensed works so long as you comply with their ASLv2 notifications, which are contained in NOTICE. Okio does not appear to specify a NOTICE, so none is needed (provided that both their source repo and the specific jar we incorporate agree).

          ref licensing howto for guidelines that agree with the above

          Show
          busbey Sean Busbey added a comment - LICENSE already contains the ASLv2 license covering the aggregate work, including Okio. You need not list the individual components of an aggregate work made up of ASLv2 licensed works so long as you comply with their ASLv2 notifications, which are contained in NOTICE. Okio does not appear to specify a NOTICE, so none is needed (provided that both their source repo and the specific jar we incorporate agree). ref licensing howto for guidelines that agree with the above
          Hide
          xiaochen Xiao Chen added a comment -

          Yep, that's the rule we used. Thanks for helping to explain, Sean.

          Show
          xiaochen Xiao Chen added a comment - Yep, that's the rule we used. Thanks for helping to explain, Sean.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the clarifications Sean and Xiao.

          Arpit Agarwal, could you edit the spreadsheet, and generate + merge the new L&N? That way we don't lose any changes during the iterations. I can also do a pass this week after you're done. Thank you.

          Hi Xiao Chen, I can do a scan by next week in case there are any more additions to NOTICE apart from jcip. Will update the spreadsheet. Am I right in understanding that L&N files are not to be edited by hand but generated completely by the Python scripts?

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the clarifications Sean and Xiao. Arpit Agarwal, could you edit the spreadsheet, and generate + merge the new L&N? That way we don't lose any changes during the iterations. I can also do a pass this week after you're done. Thank you. Hi Xiao Chen , I can do a scan by next week in case there are any more additions to NOTICE apart from jcip. Will update the spreadsheet. Am I right in understanding that L&N files are not to be edited by hand but generated completely by the Python scripts?
          Hide
          xiaochen Xiao Chen added a comment -

          Am I right in understanding that L&N files are not to be edited by hand but generated completely by the Python scripts?

          Yep, generally we update the spreadsheet, then parse it to get a new L&N. After that we'll manually merge it into existing L&Ns. This way all the information is tracked in the spreadsheet history, and individual efforts can be combined.

          If you want to do a quick edit on the L&N directly, make sure to also put it in the spreadsheet so it doesn't get overwritten by the next run. Thanks.

          Show
          xiaochen Xiao Chen added a comment - Am I right in understanding that L&N files are not to be edited by hand but generated completely by the Python scripts? Yep, generally we update the spreadsheet, then parse it to get a new L&N. After that we'll manually merge it into existing L&Ns. This way all the information is tracked in the spreadsheet history, and individual efforts can be combined. If you want to do a quick edit on the L&N directly, make sure to also put it in the spreadsheet so it doesn't get overwritten by the next run. Thanks.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          All, the latest set of comments being made here, do they block a 2.7.3 RC? If so, shall we reopen this JIRA and address them?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - All, the latest set of comments being made here, do they block a 2.7.3 RC? If so, shall we reopen this JIRA and address them?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Yes we probably need to reopen this as a blocker for 2.7.3 until we are sure there are no other missing notices.

          Show
          arpitagarwal Arpit Agarwal added a comment - Yes we probably need to reopen this as a blocker for 2.7.3 until we are sure there are no other missing notices.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          For tracking blocker issue of 2.7.3, reopening this until the confirmation ends.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited For tracking blocker issue of 2.7.3, reopening this until the confirmation ends.
          Hide
          xiaochen Xiao Chen added a comment -

          I won't be able to work on this until Friday. If anyone has cycle for the extra changes, please feel free.
          If no updates by Friday noon PDT, I will make a pass on it and update.

          Show
          xiaochen Xiao Chen added a comment - I won't be able to work on this until Friday. If anyone has cycle for the extra changes, please feel free. If no updates by Friday noon PDT, I will make a pass on it and update.
          Hide
          aw Allen Wittenauer added a comment -

          Are we going to get a documentation update so that everyone else knows how to update the licensing information with this setup? I've got YARN-5121 that needs to add some licensing info....

          Show
          aw Allen Wittenauer added a comment - Are we going to get a documentation update so that everyone else knows how to update the licensing information with this setup? I've got YARN-5121 that needs to add some licensing info....
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Allen Wittenauer how about adding a documentation of Project Dependency Management like Apache Curator? http://curator.apache.org/dependency-management.html

          Show
          ozawa Tsuyoshi Ozawa added a comment - Allen Wittenauer how about adding a documentation of Project Dependency Management like Apache Curator? http://curator.apache.org/dependency-management.html
          Hide
          aw Allen Wittenauer added a comment -

          I'm not sure I understand your suggestion. If I'm importing code from FreeBSD into our build tree, that's not a dependency problem, it's a LICENSE file problem.

          Show
          aw Allen Wittenauer added a comment - I'm not sure I understand your suggestion. If I'm importing code from FreeBSD into our build tree, that's not a dependency problem, it's a LICENSE file problem.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Ah, sorry, I misread your comment. I thought you meant we should make the spread sheet more portable to track the license information. Please ignore it.

          I think the code in YARN-5121 looks to be under 3-clause BSD license. From the ASF documentation : In LICENSE, add a pointer to the dependency's license within the distribution and a short note summarizing its licensing with version we're using.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Ah, sorry, I misread your comment. I thought you meant we should make the spread sheet more portable to track the license information. Please ignore it. I think the code in YARN-5121 looks to be under 3-clause BSD license. From the ASF documentation : In LICENSE, add a pointer to the dependency's license within the distribution and a short note summarizing its licensing with version we're using.
          Hide
          xiaochen Xiao Chen added a comment -

          Here's the addendum to address previous comments from Sean and Arpit.

          I've gone through the spreadsheet and updated what I see inconsistent. Then generated this with some manual line change etc.

          • jcip is actually CCAL, not ASL. Added L&N.
          • some deps had NOTICE which we thought don't need to include. Included them now.

          I will be on PTO until 7/5 so won't be able to follow up on this. If any findings, please feel free to update. Thanks.

          Show
          xiaochen Xiao Chen added a comment - Here's the addendum to address previous comments from Sean and Arpit. I've gone through the spreadsheet and updated what I see inconsistent. Then generated this with some manual line change etc. jcip is actually CCAL, not ASL. Added L&N. some deps had NOTICE which we thought don't need to include. Included them now. I will be on PTO until 7/5 so won't be able to follow up on this. If any findings, please feel free to update. Thanks.
          Hide
          andrew.wang Andrew Wang added a comment -

          So one thing I could use clarification is if we need to include NOTICE entries if the downstream project has incorrectly constructed their NOTICE file, or we've already captured their transitive dependencies.

          • Jetty spends a lot of space listing out their deps and licenses, but that's supposed to go in LICENSE, not NOTICE. We also already look at all of Jetty's transitive dependencies ourselves.
          • Xerces is another Apache project, so we don't need the Apache copyright. Also considering the Xerces code was donated to Apache, are they legally required to list the IBM/Sun/iClick copyright attributions?
          • None of snappy-java's entries look like legally required attributions either

          The jcip changes look good though, nice catch.

          Show
          andrew.wang Andrew Wang added a comment - So one thing I could use clarification is if we need to include NOTICE entries if the downstream project has incorrectly constructed their NOTICE file, or we've already captured their transitive dependencies. Jetty spends a lot of space listing out their deps and licenses, but that's supposed to go in LICENSE, not NOTICE. We also already look at all of Jetty's transitive dependencies ourselves. Xerces is another Apache project, so we don't need the Apache copyright. Also considering the Xerces code was donated to Apache, are they legally required to list the IBM/Sun/iClick copyright attributions? None of snappy-java's entries look like legally required attributions either The jcip changes look good though, nice catch.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew for reviewing.

          So Jetty, Xerces and snappy are all under ASFv2, and the apache guide says:
          If the dependency supplies a NOTICE file, its contents must be analyzed and the relevant portions bubbled up into the top-level NOTICE file.
          I'm unclear what exactly are the relevant portions either. Shall we file LEGAL jiras for each of them? Then we'll only include what legal considers relevant... (unless someone in the audience here has the expertise / past experience).

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew for reviewing. So Jetty, Xerces and snappy are all under ASFv2, and the apache guide says: If the dependency supplies a NOTICE file, its contents must be analyzed and the relevant portions bubbled up into the top-level NOTICE file. I'm unclear what exactly are the relevant portions either. Shall we file LEGAL jiras for each of them? Then we'll only include what legal considers relevant... (unless someone in the audience here has the expertise / past experience).
          Hide
          andrew.wang Andrew Wang added a comment -

          In the interest of "just getting it done", I'm okay committing this for now and then fixing it later based on LEGAL input.

          Xiao, do you mind filing a new JIRA with this addendum patch? I'll +1 and commit it. If these changes also apply to the branch-2's, LMK via the target versions.

          I'll also file the LEGAL JIRA and link it here, since this seems like our umbrella JIRA for L&N changes.

          Show
          andrew.wang Andrew Wang added a comment - In the interest of "just getting it done", I'm okay committing this for now and then fixing it later based on LEGAL input. Xiao, do you mind filing a new JIRA with this addendum patch? I'll +1 and commit it. If these changes also apply to the branch-2's, LMK via the target versions. I'll also file the LEGAL JIRA and link it here, since this seems like our umbrella JIRA for L&N changes.
          Hide
          andrew.wang Andrew Wang added a comment -

          Filed NOTICE questions over at LEGAL-262.

          Show
          andrew.wang Andrew Wang added a comment - Filed NOTICE questions over at LEGAL-262 .
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew for the suggestion. I created HADOOP-13350 and attached the addendum patches there.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew for the suggestion. I created HADOOP-13350 and attached the addendum patches there.
          Hide
          busbey Sean Busbey added a comment -

          Should this jira also have a fixVersion of 2.8.0 (and 3.0.0-alpha1?), since there isn't necessarily a relationship between the release of 2.7.3 and 2.8.0?

          Show
          busbey Sean Busbey added a comment - Should this jira also have a fixVersion of 2.8.0 (and 3.0.0-alpha1?), since there isn't necessarily a relationship between the release of 2.7.3 and 2.8.0?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Given 2.7.3 will be released earlier than 2.8.0, we don't need to set the version of 2.8.0 or 3.0.0-alpha1. (e.g. A fix in 2.7.3 is always in 2.8.0, so we don't need to set 2.8.0 explicitly.)

          Show
          ajisakaa Akira Ajisaka added a comment - Given 2.7.3 will be released earlier than 2.8.0, we don't need to set the version of 2.8.0 or 3.0.0-alpha1. (e.g. A fix in 2.7.3 is always in 2.8.0, so we don't need to set 2.8.0 explicitly.)
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Closing this. We need to fix HADOOP-13312 and HADOOP-13298.

          Show
          ajisakaa Akira Ajisaka added a comment - Closing this. We need to fix HADOOP-13312 and HADOOP-13298 .
          Hide
          busbey Sean Busbey added a comment -

          That is very confusing, because 2.7.1+ and 2.8.0+ are different release trains. Neither release is out yet, and any number of issues could cause 2.7.3 to come out after 2.8.0. It also requires downstream folks to know about actual release dates of maintenance releases to know if an issue they care about was fixed in the new minor version they want to use. Consider, for example, someone on Hadoop 2.6.z (or even 2.4.z) who is considering upgrading to 2.8.0. If some issue fixed in "2.7.3" is important to them, they have to either go to git history or look up release dates.

          Show
          busbey Sean Busbey added a comment - That is very confusing, because 2.7.1+ and 2.8.0+ are different release trains. Neither release is out yet, and any number of issues could cause 2.7.3 to come out after 2.8.0. It also requires downstream folks to know about actual release dates of maintenance releases to know if an issue they care about was fixed in the new minor version they want to use. Consider, for example, someone on Hadoop 2.6.z (or even 2.4.z) who is considering upgrading to 2.8.0. If some issue fixed in "2.7.3" is important to them, they have to either go to git history or look up release dates.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          According to https://wiki.apache.org/hadoop/HowToCommit,

          Always set the "Fix Version" at this point, but please only set a single fix version, the earliest release in which the change will appear.

          so I thought it's okay to set only "2.7.3" and "2.6.5".

          That being said, I agreed with you that it's worth adding 2.8.0. I'm thinking we need to discuss it on dev@.

          Show
          ajisakaa Akira Ajisaka added a comment - According to https://wiki.apache.org/hadoop/HowToCommit , Always set the "Fix Version" at this point, but please only set a single fix version, the earliest release in which the change will appear. so I thought it's okay to set only "2.7.3" and "2.6.5". That being said, I agreed with you that it's worth adding 2.8.0. I'm thinking we need to discuss it on dev@.
          Hide
          busbey Sean Busbey added a comment -

          Yeah, definitely I think this needs clarification. Under the interpretation you're using, I don't see why you wouldn't just choose one of 2.7.3 or 2.6.5 depending on which will get released first.

          Show
          busbey Sean Busbey added a comment - Yeah, definitely I think this needs clarification. Under the interpretation you're using, I don't see why you wouldn't just choose one of 2.7.3 or 2.6.5 depending on which will get released first.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          This is a special case.

          Special case- when committing to a non-mainline branch (such as branch-0.22 or branch-0.23 ATM), please set fix-version to either 2.x.x or 3.x.x appropriately too.

          I'm thinking 2.7.3 will be released first, so we need to add 2.7.3. In addition, patches committed in branch-2.7 are not always committed in branch-2.6, so we need to add 2.6.5 explicitly if the patch is committed in branch-2.6.

          Show
          ajisakaa Akira Ajisaka added a comment - This is a special case. Special case- when committing to a non-mainline branch (such as branch-0.22 or branch-0.23 ATM), please set fix-version to either 2.x.x or 3.x.x appropriately too. I'm thinking 2.7.3 will be released first, so we need to add 2.7.3. In addition, patches committed in branch-2.7 are not always committed in branch-2.6, so we need to add 2.6.5 explicitly if the patch is committed in branch-2.6.
          Hide
          andrew.wang Andrew Wang added a comment -

          LEGAL-262 has been resolved, consensus was to just use the NOTICE files as is (which is what we already did). So I think we're good on that front.

          Show
          andrew.wang Andrew Wang added a comment - LEGAL-262 has been resolved, consensus was to just use the NOTICE files as is (which is what we already did). So I think we're good on that front.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development