Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11656 Classpath isolation for downstream clients
  3. HADOOP-11804

Shaded Hadoop client artifacts and minicluster

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha2
    • build
    • None
    • Hide
      The `hadoop-client` Maven artifact available in 2.x releases pulls
      Hadoop's transitive dependencies onto a Hadoop application's classpath.
      This can be problematic if the versions of these transitive dependencies
      conflict with the versions used by the application.

      HADOOP-11804 adds new `hadoop-client-api` and
      `hadoop-client-runtime` artifacts that shade Hadoop's dependencies
      into a single jar. This avoids leaking Hadoop's dependencies onto the
      application's classpath.
      Show
      The `hadoop-client` Maven artifact available in 2.x releases pulls Hadoop's transitive dependencies onto a Hadoop application's classpath. This can be problematic if the versions of these transitive dependencies conflict with the versions used by the application. HADOOP-11804 adds new `hadoop-client-api` and `hadoop-client-runtime` artifacts that shade Hadoop's dependencies into a single jar. This avoids leaking Hadoop's dependencies onto the application's classpath.

    Description

      make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to talk with a Hadoop cluster without seeing any of the implementation dependencies.

      see proposal on parent for details.

      Attachments

        1. HADOOP-11804.1.patch
          76 kB
          Sean Busbey
        2. HADOOP-11804.10.patch
          102 kB
          Sean Busbey
        3. HADOOP-11804.11.patch
          121 kB
          Sean Busbey
        4. HADOOP-11804.12.patch
          120 kB
          Sean Busbey
        5. HADOOP-11804.13.patch
          121 kB
          Sean Busbey
        6. HADOOP-11804.14.patch
          122 kB
          Sean Busbey
        7. HADOOP-11804.2.patch
          76 kB
          Sean Busbey
        8. HADOOP-11804.3.patch
          82 kB
          Sean Busbey
        9. HADOOP-11804.4.patch
          86 kB
          Sean Busbey
        10. HADOOP-11804.5.patch
          91 kB
          Sean Busbey
        11. HADOOP-11804.6.patch
          96 kB
          Sean Busbey
        12. HADOOP-11804.7.patch
          139 kB
          Sean Busbey
        13. HADOOP-11804.8.patch
          103 kB
          Sean Busbey
        14. HADOOP-11804.9.patch
          103 kB
          Sean Busbey
        15. hadoop-11804-client-test.tar.gz
          1 kB
          Sean Busbey

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-13977 IntelliJ Compilation error in ITUseMiniCluster.java

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-13922 Some modules have dependencies on hadoop-client jar removed by HADOOP-11804

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        is blocked by

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-13789 Hadoop Common includes generated test protos in both jar and test-jar

        • Major - Major loss of function.
        • Resolved
        is depended upon by

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-33212 Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

        • Major - Major loss of function.
        • Resolved
        is duplicated by

        Sub-task - The sub-task of the issue HADOOP-13136 shade protobuf in the hadoop-common jar

        • Major - Major loss of function.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue HADOOP-13917 Ensure yetus personality runs the integration tests for the shaded client

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Sub-task - The sub-task of the issue HADOOP-13921 Remove Log4j classes from JobConf

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-14284 Shade Guava everywhere

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. YARN-6466 Provide shaded framework jar for containers

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13916 Document how downstream clients should make use of the new shaded client artifacts

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13918 Add integration tests for shaded client based on use by HBase

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13919 add integration tests for shaded client based on use by Spark

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16822 Provide source artifacts for hadoop-client-api

        • Major - Major loss of function.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-12048 many undeclared used dependencies (and declared unused dependencies)

        • Major - Major loss of function.
        • Open
        supercedes

        Sub-task - The sub-task of the issue HADOOP-13136 shade protobuf in the hadoop-common jar

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            busbey Sean Busbey
            busbey Sean Busbey
            Votes:
            1 Vote for this issue
            Watchers:
            40 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment