Hadoop Common
  1. Hadoop Common
  2. HADOOP-9407

commons-daemon 1.0.3 dependency has bad group id causing build issues

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta, 2.0.5-alpha
    • Component/s: build
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      The commons-daemon dependency of the hadoop-hdfs module has been at version 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in its pom ("org.apache.commons" as opposed to "commons-daemon"). This problem has since been corrected on commons-daemon starting 1.0.4.

      This causes build problems for many who depend on hadoop-hdfs directly and indirectly, however. Maven can skip over this metadata inconsistency. But other less forgiving build systems such as ivy and gradle have much harder time working around this problem. For example, in gradle, pretty much the only obvious way to work around this is to override this dependency version.

      1. HDFS-4497.patch
        0.5 kB
        Sangjin Lee

        Activity

        Show
        Sangjin Lee added a comment - Some links on the build problems: https://issues.sonatype.org/browse/MVNCENTRAL-58 http://mail-archives.apache.org/mod_mbox/ant-ivy-user/201103.mbox/%3C357652.85978.qm@web30807.mail.mud.yahoo.com%3E http://markmail.org/thread/2of7xjzrir5lurmi
        Hide
        Sangjin Lee added a comment -

        patch submitted to upgrade commons-daemon from 1.0.3 to 1.0.13

        Show
        Sangjin Lee added a comment - patch submitted to upgrade commons-daemon from 1.0.3 to 1.0.13
        Hide
        Steve Loughran added a comment -

        -do you know what changes are there between the versions?

        The release notes - http://www.apache.org/dist/commons/daemon/RELEASE-NOTES.txt - list features, but the bug list doesn't say if anything is incompatible or not.

        Show
        Steve Loughran added a comment - -do you know what changes are there between the versions? The release notes - http://www.apache.org/dist/commons/daemon/RELEASE-NOTES.txt - list features, but the bug list doesn't say if anything is incompatible or not.
        Hide
        Sangjin Lee added a comment -

        I don't have that information, other than I was able to build hadoop-hdfs cleanly and all unit tests pass with the new version. I can dig some more information on the differences between 1.0.3 and 1.0.13.

        Show
        Sangjin Lee added a comment - I don't have that information, other than I was able to build hadoop-hdfs cleanly and all unit tests pass with the new version. I can dig some more information on the differences between 1.0.3 and 1.0.13.
        Hide
        Sangjin Lee added a comment -

        To be fair, the release notes do say 1.0.13 is binary-, source-, and semantic-compatible with 1.0.3.

        Show
        Sangjin Lee added a comment - To be fair, the release notes do say 1.0.13 is binary-, source-, and semantic-compatible with 1.0.3.
        Hide
        Chris Nauroth added a comment -

        A side note: commons-daemon stopped distributing pre-compiled packages for the native code for all platforms except Windows after version 1.0.5. This includes Linux.

        On branch-1, the build.xml contains code to bundle the Linux native package, so we wouldn't be able to backport this change to branch-1, at least not in a backwards-compatible way. We'd have to advise users to build the native code themselves.

        I couldn't find equivalent build code on trunk for bundling the Linux native bits, so maybe this is not a concern for trunk.

        http://mail-archives.apache.org/mod_mbox/commons-user/201302.mbox/%3CCAOGo0Vbgi%2BrDVm67qjU0XLJiUEN-FZhAfujPLKSqyO20Tq7uXQ%40mail.gmail.com%3E

        http://archive.apache.org/dist/commons/daemon/binaries/

        Show
        Chris Nauroth added a comment - A side note: commons-daemon stopped distributing pre-compiled packages for the native code for all platforms except Windows after version 1.0.5. This includes Linux. On branch-1, the build.xml contains code to bundle the Linux native package, so we wouldn't be able to backport this change to branch-1, at least not in a backwards-compatible way. We'd have to advise users to build the native code themselves. I couldn't find equivalent build code on trunk for bundling the Linux native bits, so maybe this is not a concern for trunk. http://mail-archives.apache.org/mod_mbox/commons-user/201302.mbox/%3CCAOGo0Vbgi%2BrDVm67qjU0XLJiUEN-FZhAfujPLKSqyO20Tq7uXQ%40mail.gmail.com%3E http://archive.apache.org/dist/commons/daemon/binaries/
        Hide
        Sangjin Lee added a comment -

        I do not believe the new version needs to be 1.0.13. Another possibility is to go to 1.0.5, which may minimize changes on the hadoop side. Thoughts?

        Show
        Sangjin Lee added a comment - I do not believe the new version needs to be 1.0.13. Another possibility is to go to 1.0.5, which may minimize changes on the hadoop side. Thoughts?
        Hide
        Chris Nauroth added a comment -

        I just found some more historical background on this. In HDFS-2303, the community made a specific choice to stop bundling jsvc on branch-2 and later. branch-1 continues to bundle it though. Considering that, I don't think we necessarily have a requirement to keep the commons-daemon version in sync between trunk and branch-1. Therefore, I think it's OK to upgrade commons-daemon to latest (1.0.13) on trunk as long as we're confident about testing and compatibility.

        Show
        Chris Nauroth added a comment - I just found some more historical background on this. In HDFS-2303 , the community made a specific choice to stop bundling jsvc on branch-2 and later. branch-1 continues to bundle it though. Considering that, I don't think we necessarily have a requirement to keep the commons-daemon version in sync between trunk and branch-1. Therefore, I think it's OK to upgrade commons-daemon to latest (1.0.13) on trunk as long as we're confident about testing and compatibility.
        Hide
        Mladen Turk added a comment -

        The 1.0.3 and 1.0.13 should be compatible. The API didn't change, at least for the part you guys are using.
        We added bunch of new features, but ensuring we don't mess with backward compatibility.

        The reason for not shipping Linux binaries is the fact that we switched to more secure libcap, and that lib can have different .so names across different Linux distros, so we cannot have a generic one. However many distros already provide daemon binaries, and for the ones that don't the build is simple 'configure && make install'.

        Looking at the
        branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
        I see couple of things that can be improved, but basically that's fine and will work with 1.0.13. In any case eventual fix would be trivial since you don't use singnaling (SIG_USR2) API inside SecureDataNodeStarter, and I'm ready to help you guys with that.
        If you have have some nightly (or something like that) binaries I do a couple of smoke test and come up with eventual fixes.

        Like said I'd suggest you upgrade to more recent versions mainly for security reasons and the fact that many linux distros already ship with contemporary versions.

        Show
        Mladen Turk added a comment - The 1.0.3 and 1.0.13 should be compatible. The API didn't change, at least for the part you guys are using. We added bunch of new features, but ensuring we don't mess with backward compatibility. The reason for not shipping Linux binaries is the fact that we switched to more secure libcap, and that lib can have different .so names across different Linux distros, so we cannot have a generic one. However many distros already provide daemon binaries, and for the ones that don't the build is simple 'configure && make install'. Looking at the branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs I see couple of things that can be improved, but basically that's fine and will work with 1.0.13. In any case eventual fix would be trivial since you don't use singnaling (SIG_USR2) API inside SecureDataNodeStarter, and I'm ready to help you guys with that. If you have have some nightly (or something like that) binaries I do a couple of smoke test and come up with eventual fixes. Like said I'd suggest you upgrade to more recent versions mainly for security reasons and the fact that many linux distros already ship with contemporary versions.
        Hide
        Sangjin Lee added a comment -

        HDFS folks, are you OK with the patch? Thanks!

        Show
        Sangjin Lee added a comment - HDFS folks, are you OK with the patch? Thanks!
        Hide
        Chris Nauroth added a comment -

        +1 for the current patch, upgrading to 1.0.13.

        BTW, I've been doing some testing of my own with 1.0.13 for an unrelated reason, and it's working fine.

        Show
        Chris Nauroth added a comment - +1 for the current patch, upgrading to 1.0.13. BTW, I've been doing some testing of my own with 1.0.13 for an unrelated reason, and it's working fine.
        Hide
        Sangjin Lee added a comment -

        Ping?

        Show
        Sangjin Lee added a comment - Ping?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569284/HDFS-4497.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4037//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4037//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569284/HDFS-4497.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4037//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4037//console This message is automatically generated.
        Hide
        Sangjin Lee added a comment -

        Since the patch is about updating an external dependency, no new tests are needed. It'd be great if someone could give a +1 and commit the patch. Thanks!

        Show
        Sangjin Lee added a comment - Since the patch is about updating an external dependency, no new tests are needed. It'd be great if someone could give a +1 and commit the patch. Thanks!
        Hide
        Lohit Vijayarenu added a comment -

        +1 Patch looks good

        Show
        Lohit Vijayarenu added a comment - +1 Patch looks good
        Hide
        Suresh Srinivas added a comment -

        Sangjin Lee if you file an ICLA, then I can add you as a contributor and assign this jira to you. I will also commit this patch, once ICLA is filed. Please see - http://www.apache.org/licenses/icla.txt.

        Show
        Suresh Srinivas added a comment - Sangjin Lee if you file an ICLA, then I can add you as a contributor and assign this jira to you. I will also commit this patch, once ICLA is filed. Please see - http://www.apache.org/licenses/icla.txt .
        Hide
        Sangjin Lee added a comment -

        Suresh Srinivas I am already an Apache committer (mina): http://people.apache.org/committer-index.html#sjlee

        I also filed the ICLA a long time back before I became a committer.

        Show
        Sangjin Lee added a comment - Suresh Srinivas I am already an Apache committer (mina): http://people.apache.org/committer-index.html#sjlee I also filed the ICLA a long time back before I became a committer.
        Hide
        Sangjin Lee added a comment -

        Suresh, would you be then able to assign this to me so I can commit?

        Show
        Sangjin Lee added a comment - Suresh, would you be then able to assign this to me so I can commit?
        Hide
        Suresh Srinivas added a comment -

        Sangjin Lee Added you as a contributor and assigned the jira to you. I do not think you will be able to commit the patch though.

        I committed the patch to branch-2 and trunk. Thank you Sangjin.

        Show
        Suresh Srinivas added a comment - Sangjin Lee Added you as a contributor and assigned the jira to you. I do not think you will be able to commit the patch though. I committed the patch to branch-2 and trunk. Thank you Sangjin.
        Hide
        Suresh Srinivas added a comment -

        Also thanks to Lohit for the review and testing.

        Show
        Suresh Srinivas added a comment - Also thanks to Lohit for the review and testing.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3475 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3475/)
        HADOOP-9407. commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3475 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3475/ ) HADOOP-9407 . commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Sangjin Lee added a comment -
        Show
        Sangjin Lee added a comment - Suresh Srinivas Thanks!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #156 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/156/)
        HADOOP-9407. commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #156 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/156/ ) HADOOP-9407 . commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1345 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1345/)
        HADOOP-9407. commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704)

        Result = FAILURE
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1345 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1345/ ) HADOOP-9407 . commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1373 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1373/)
        HADOOP-9407. commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704)

        Result = FAILURE
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1373 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1373/ ) HADOOP-9407 . commons-daemon 1.0.3 dependency has bad group id causing build issues. Contributed by Sangjin Lee. (Revision 1456704) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1456704 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Konstantin Boudnik added a comment -

        I think it needs to be added to branch-2.0.4-alpha as well before the release goes out.

        Show
        Konstantin Boudnik added a comment - I think it needs to be added to branch-2.0.4-alpha as well before the release goes out.
        Hide
        Sangjin Lee added a comment -

        Konstantin Boudnik +1. What needs to be done for that to happen? I raised it during the first voting for 2.0.4-alpha. Thanks!

        Show
        Sangjin Lee added a comment - Konstantin Boudnik +1. What needs to be done for that to happen? I raised it during the first voting for 2.0.4-alpha. Thanks!
        Hide
        Konstantin Boudnik added a comment -

        I have raised it again on the voting thread with a neutral vote (-0). For a variety of reasons I am not very optimistic that RC will be re-spun at this point.

        Show
        Konstantin Boudnik added a comment - I have raised it again on the voting thread with a neutral vote (-0). For a variety of reasons I am not very optimistic that RC will be re-spun at this point.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569284/HDFS-4497.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2466//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569284/HDFS-4497.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2466//console This message is automatically generated.
        Hide
        Sangjin Lee added a comment -

        The patch has already been applied to trunk (and to branch-2), thus the above patch error. I just flipped the status back to "patch available" which seemed to have triggered the build.

        Show
        Sangjin Lee added a comment - The patch has already been applied to trunk (and to branch-2), thus the above patch error. I just flipped the status back to "patch available" which seemed to have triggered the build.
        Hide
        Konstantin Boudnik added a comment -

        I have reopened to ticket to be able to back-ported to branch-2.0.4-alpha. "Patch available" wasn't needed.

        Show
        Konstantin Boudnik added a comment - I have reopened to ticket to be able to back-ported to branch-2.0.4-alpha. "Patch available" wasn't needed.
        Hide
        Sangjin Lee added a comment -

        Thanks for the clarification...

        Show
        Sangjin Lee added a comment - Thanks for the clarification...
        Hide
        Konstantin Boudnik added a comment -

        I have just committed this fix to 2.0.4.1-alpha release (branch-2.0.4-alpha). Thank you Sangjin.

        Show
        Konstantin Boudnik added a comment - I have just committed this fix to 2.0.4.1-alpha release (branch-2.0.4-alpha). Thank you Sangjin.
        Hide
        Sangjin Lee added a comment -

        Thanks Konstantin!

        Show
        Sangjin Lee added a comment - Thanks Konstantin!
        Hide
        Konstantin Boudnik added a comment -

        I see this already committed to the branch-2, which is in my understanding is effectively tracking 2.0.5 release now. So, I guess the ticket should be closed.

        Show
        Konstantin Boudnik added a comment - I see this already committed to the branch-2, which is in my understanding is effectively tracking 2.0.5 release now. So, I guess the ticket should be closed.
        Hide
        Konstantin Boudnik added a comment -

        Fixed in both targetted releases. CLosing.

        Show
        Konstantin Boudnik added a comment - Fixed in both targetted releases. CLosing.

          People

          • Assignee:
            Sangjin Lee
            Reporter:
            Sangjin Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development