Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11656 Classpath isolation for downstream clients
  3. HADOOP-11804

Shaded Hadoop client artifacts and minicluster

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha2
    • build
    • None
    • Hide
      The `hadoop-client` Maven artifact available in 2.x releases pulls
      Hadoop's transitive dependencies onto a Hadoop application's classpath.
      This can be problematic if the versions of these transitive dependencies
      conflict with the versions used by the application.

      HADOOP-11804 adds new `hadoop-client-api` and
      `hadoop-client-runtime` artifacts that shade Hadoop's dependencies
      into a single jar. This avoids leaking Hadoop's dependencies onto the
      application's classpath.
      Show
      The `hadoop-client` Maven artifact available in 2.x releases pulls Hadoop's transitive dependencies onto a Hadoop application's classpath. This can be problematic if the versions of these transitive dependencies conflict with the versions used by the application. HADOOP-11804 adds new `hadoop-client-api` and `hadoop-client-runtime` artifacts that shade Hadoop's dependencies into a single jar. This avoids leaking Hadoop's dependencies onto the application's classpath.

    Description

      make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to talk with a Hadoop cluster without seeing any of the implementation dependencies.

      see proposal on parent for details.

      Attachments

        1. HADOOP-11804.1.patch
          76 kB
          Sean Busbey
        2. HADOOP-11804.10.patch
          102 kB
          Sean Busbey
        3. HADOOP-11804.11.patch
          121 kB
          Sean Busbey
        4. HADOOP-11804.12.patch
          120 kB
          Sean Busbey
        5. HADOOP-11804.13.patch
          121 kB
          Sean Busbey
        6. HADOOP-11804.14.patch
          122 kB
          Sean Busbey
        7. HADOOP-11804.2.patch
          76 kB
          Sean Busbey
        8. HADOOP-11804.3.patch
          82 kB
          Sean Busbey
        9. HADOOP-11804.4.patch
          86 kB
          Sean Busbey
        10. HADOOP-11804.5.patch
          91 kB
          Sean Busbey
        11. HADOOP-11804.6.patch
          96 kB
          Sean Busbey
        12. HADOOP-11804.7.patch
          139 kB
          Sean Busbey
        13. HADOOP-11804.8.patch
          103 kB
          Sean Busbey
        14. HADOOP-11804.9.patch
          103 kB
          Sean Busbey
        15. hadoop-11804-client-test.tar.gz
          1 kB
          Sean Busbey

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13977 IntelliJ Compilation error in ITUseMiniCluster.java

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13922 Some modules have dependencies on hadoop-client jar removed by HADOOP-11804

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13789 Hadoop Common includes generated test protos in both jar and test-jar

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-33212 Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue HADOOP-13136 shade protobuf in the hadoop-common jar

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-13917 Ensure yetus personality runs the integration tests for the shaded client

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13921 Remove Log4j classes from JobConf

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14284 Shade Guava everywhere

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-6466 Provide shaded framework jar for containers

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13916 Document how downstream clients should make use of the new shaded client artifacts

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13918 Add integration tests for shaded client based on use by HBase

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13919 add integration tests for shaded client based on use by Spark

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16822 Provide source artifacts for hadoop-client-api

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-12048 many undeclared used dependencies (and declared unused dependencies)

          • Major - Major loss of function.
          • Open
          supercedes

          Sub-task - The sub-task of the issue HADOOP-13136 shade protobuf in the hadoop-common jar

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              busbey Sean Busbey
              busbey Sean Busbey
              Votes:
              1 Vote for this issue
              Watchers:
              39 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: