Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-1161

Update hadoop dependencies to Hadoop 3

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • Build
    • None

    Description

      With the release of Hadoop 3 the upcoming release of Knox 1.0.0 should use Hadoop 3 dependencies.

      Attachments

        1. KNOX-1161.001.patch
          7 kB
          Sandeep More
        2. KNOX-1161-revised.patch
          4 kB
          Colm O hEigeartaigh
        3. KNOX-1161-revised.patch.2
          6 kB
          Colm O hEigeartaigh

        Issue Links

          Activity

            In my opinion, we need to be very careful about any technology upgrades this late in the game. It would be wise to at least do testing against Hadoop 3 though to identify incompatibilities, etc. I would think we want Knox 1.0.0 to work again both 2.x and 3.x if at all possible.

            rkellogg Richard Kellog Jr. added a comment - In my opinion, we need to be very careful about any technology upgrades this late in the game. It would be wise to at least do testing against Hadoop 3 though to identify incompatibilities, etc. I would think we want Knox 1.0.0 to work again both 2.x and 3.x if at all possible.
            smore Sandeep More added a comment -

            Right, although this bug is about upgrading the libraries that Knox relies on to build and pull in features such as Hadoop Group Lookups . This would not change the way Knox interfaces with other components so Knox should work with Hadoop 2.x just as fine as it worked with Hadoop 3.x before this.

            smore Sandeep More added a comment - Right, although this bug is about upgrading the libraries that Knox relies on to build and pull in features such as Hadoop Group Lookups . This would not change the way Knox interfaces with other components so Knox should work with Hadoop 2.x just as fine as it worked with Hadoop 3.x before this.

            Commit 772bc33d48dd37be5cd098992df3e50db24326f3 in knox's branch refs/heads/master from moresandeep
            [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=772bc33 ]

            KNOX-1161 - Update hadoop dependencies to hadoop 3

            jira-bot ASF subversion and git services added a comment - Commit 772bc33d48dd37be5cd098992df3e50db24326f3 in knox's branch refs/heads/master from moresandeep [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=772bc33 ] KNOX-1161 - Update hadoop dependencies to hadoop 3
            lmccay Larry McCay added a comment -

            +1 to smore's assertion here.

            We do not have the same sort of compatibility issues that others do which require shims and the like.
            Our use of the hadoop common functionality doesn't really have wire level compatibility issues like APIs.

            The Hadoop Auth Provider can leverage the Hadoop 3 facilities without having a compatibility issue with Hadoop 2.x clusters - as the authentication is encapsulated within the Knox process itself and identity is typically asserted via doas/user.name params - which remains compatible.

            Same dynamic for the Hadoop Groups Mapping integration in the Hadoop Group Lookup Provider.

            I also know of downstream testing of Knox 0.14.0 with Hadoop 3 dependencies.
            We should be fine.

            Also, I think the separation of 0.14.0 and 1.0.0 on the Hadoop version boundary makes a lot of sense.
            Functionality will be largely the same therefore one can put off uptake of 1.0.0 until there is comfort with Hadoop 3 migration - if so desired.

            lmccay Larry McCay added a comment - +1 to smore 's assertion here. We do not have the same sort of compatibility issues that others do which require shims and the like. Our use of the hadoop common functionality doesn't really have wire level compatibility issues like APIs. The Hadoop Auth Provider can leverage the Hadoop 3 facilities without having a compatibility issue with Hadoop 2.x clusters - as the authentication is encapsulated within the Knox process itself and identity is typically asserted via doas/user.name params - which remains compatible. Same dynamic for the Hadoop Groups Mapping integration in the Hadoop Group Lookup Provider. I also know of downstream testing of Knox 0.14.0 with Hadoop 3 dependencies. We should be fine. Also, I think the separation of 0.14.0 and 1.0.0 on the Hadoop version boundary makes a lot of sense. Functionality will be largely the same therefore one can put off uptake of 1.0.0 until there is comfort with Hadoop 3 migration - if so desired.
            smore Sandeep More added a comment -

            Reopening the bug to address the concerns raised by coheigea

            smore Sandeep More added a comment - Reopening the bug to address the concerns raised by coheigea

            Hi moresandeep,

            Please see attached for some proposed changes (feel free to ignore any of them). The changes are:

            • Re-used the core Mockito/Jackson versions in gateway-test-release for consistency
            • Use the Hadoop 3.0.0 minikdc in gateway-test-release instead of the alpha version, and remove the dependency on Kerby.
            • Remove unused "jackson2.version" declaration in root pom
            • Remove Kerby exclusions and subsequent dependency in the root pom.

            All tests pass with this patch. Let me know what you think!

            BTW there are two outstanding issues I noticed with the distribution:

            a) We are now shipping all of the kerby jars...probably these should be excluded as I don't think they're needed by Knox.
            b) We're also shipping re2j with is BSD 3-clause...we need to acknowledge this as such in our LICENSE/NOTICE.

            coheigea Colm O hEigeartaigh added a comment - Hi moresandeep , Please see attached for some proposed changes (feel free to ignore any of them). The changes are: Re-used the core Mockito/Jackson versions in gateway-test-release for consistency Use the Hadoop 3.0.0 minikdc in gateway-test-release instead of the alpha version, and remove the dependency on Kerby. Remove unused "jackson2.version" declaration in root pom Remove Kerby exclusions and subsequent dependency in the root pom. All tests pass with this patch. Let me know what you think! BTW there are two outstanding issues I noticed with the distribution: a) We are now shipping all of the kerby jars...probably these should be excluded as I don't think they're needed by Knox. b) We're also shipping re2j with is BSD 3-clause...we need to acknowledge this as such in our LICENSE/NOTICE.
            smore Sandeep More added a comment -

            Thanks coheigea The patch looks great, I was able to build it successfully.
            Since you provided the patch please feel free to commit it

            smore Sandeep More added a comment - Thanks coheigea The patch looks great, I was able to build it successfully. Since you provided the patch please feel free to commit it

            An update to the previous patch to add the RE2J license + exclude the Kerby jars from the distribution deps.

            coheigea Colm O hEigeartaigh added a comment - An update to the previous patch to add the RE2J license + exclude the Kerby jars from the distribution deps.

            Thanks moresandeep, I updated the patch to fix the license issue + exclude the Kerby deps from the distribution, please take a look when you get a chance + I will commit it then.

            coheigea Colm O hEigeartaigh added a comment - Thanks moresandeep , I updated the patch to fix the license issue + exclude the Kerby deps from the distribution, please take a look when you get a chance + I will commit it then.
            smore Sandeep More added a comment -

            +1 !!
            Looks good, and thanks for RE2J license and exclusion of Kerby jars.

            smore Sandeep More added a comment - +1 !! Looks good, and thanks for RE2J license and exclusion of Kerby jars.

            Commit 99e6a54afb53fdd8b4311f9a5892698d2551c635 in knox's branch refs/heads/master from coheigea
            [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=99e6a54 ]

            KNOX-1161 - Update hadoop dependencies to Hadoop 3 (Colm O hEigeartaigh, reviewed by Sandeep More)

            jira-bot ASF subversion and git services added a comment - Commit 99e6a54afb53fdd8b4311f9a5892698d2551c635 in knox's branch refs/heads/master from coheigea [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=99e6a54 ] KNOX-1161 - Update hadoop dependencies to Hadoop 3 (Colm O hEigeartaigh, reviewed by Sandeep More)

            Commit 772bc33d48dd37be5cd098992df3e50db24326f3 in knox's branch refs/heads/KNOX-998-Package_Restructuring from moresandeep
            [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=772bc33 ]

            KNOX-1161 - Update hadoop dependencies to hadoop 3

            jira-bot ASF subversion and git services added a comment - Commit 772bc33d48dd37be5cd098992df3e50db24326f3 in knox's branch refs/heads/ KNOX-998 -Package_Restructuring from moresandeep [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=772bc33 ] KNOX-1161 - Update hadoop dependencies to hadoop 3

            Commit 99e6a54afb53fdd8b4311f9a5892698d2551c635 in knox's branch refs/heads/KNOX-998-Package_Restructuring from coheigea
            [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=99e6a54 ]

            KNOX-1161 - Update hadoop dependencies to Hadoop 3 (Colm O hEigeartaigh, reviewed by Sandeep More)

            jira-bot ASF subversion and git services added a comment - Commit 99e6a54afb53fdd8b4311f9a5892698d2551c635 in knox's branch refs/heads/ KNOX-998 -Package_Restructuring from coheigea [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=99e6a54 ] KNOX-1161 - Update hadoop dependencies to Hadoop 3 (Colm O hEigeartaigh, reviewed by Sandeep More)

            People

              smore Sandeep More
              smore Sandeep More
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: