Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: 3.0.0, 2.1.0
    • Component/s: mapreduce, shading
    • Labels:
      None

      Description

      AFAICT, we should just entirely skip including hadoop in our shaded mapreduce module

      1) Folks expect to run yarn / mr apps via hadoop jar / yarn jar
      2) those commands include all the needed Hadoop jars in your classpath by default (both client side and in the containers)
      3) If you try to use "user classpath first" for your job as a workaround (e.g. for some library your application needs that hadoop provides) then our inclusion of some but not all hadoop classes then causes everything to fall over because of mixing rewritten and non-rewritten hadoop classes
      4) if you don't use "user classpath first" then all of our non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just wasting space

        Attachments

        1. HBASE-20332.3.patch
          102 kB
          Sean Busbey
        2. HBASE-20332.2.WIP.patch
          85 kB
          Sean Busbey
        3. HBASE-20332.1.WIP.patch
          82 kB
          Sean Busbey
        4. HBASE-20332.0.patch
          37 kB
          Sean Busbey

          Activity

            People

            • Assignee:
              busbey Sean Busbey
              Reporter:
              busbey Sean Busbey
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: