Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Build System
    • Labels:
      None

      Description

      As a first step towards FLINK-2268, I would suggest to put all hadoop dependencies into a jar separate from Flink's fat jar.

      This would allow users to put a custom Hadoop jar in there, or even deploy Flink without a Hadoop fat jar at all in environments where Hadoop is provided (EMR).

        Issue Links

          Activity

          Hide
          rmetzger Robert Metzger added a comment -

          Haohui Mai Thanks a lot for assigning that issue to you.
          If we are lucky, this change is relatively easy to do and requires just some changes on the shading / assembly plugin.
          Please let me know if there's anything that's unclear so that we can discuss it.

          Show
          rmetzger Robert Metzger added a comment - Haohui Mai Thanks a lot for assigning that issue to you. If we are lucky, this change is relatively easy to do and requires just some changes on the shading / assembly plugin. Please let me know if there's anything that's unclear so that we can discuss it.
          Hide
          rmetzger Robert Metzger added a comment -

          Haohui Mai what's the status on this issue?

          Show
          rmetzger Robert Metzger added a comment - Haohui Mai what's the status on this issue?
          Hide
          wheat9 Haohui Mai added a comment -

          Sorry for the delay I'm just busy on other issues – I'll take care of it in a day or two.

          Show
          wheat9 Haohui Mai added a comment - Sorry for the delay I'm just busy on other issues – I'll take care of it in a day or two.
          Hide
          rmetzger Robert Metzger added a comment -

          Okay, thank you.

          Show
          rmetzger Robert Metzger added a comment - Okay, thank you.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user haohui opened a pull request:

          https://github.com/apache/flink/pull/3604

          FLINK-5998 Un-fat Hadoop from Flink fat jar

          This PR implements FLINK-5998.

          It marks all Hadoop dependency in the dist jar as `provided` so that users can plug in the jars from their own Hadoop distribution.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/haohui/flink FLINK-5998

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3604.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3604


          commit a8f145cce141450edeb92c5ea9bed6d0441dc94f
          Author: Haohui Mai <wheat9@apache.org>
          Date: 2017-03-23T21:46:45Z

          FLINK-5998 Un-fat Hadoop from Flink fat jar.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user haohui opened a pull request: https://github.com/apache/flink/pull/3604 FLINK-5998 Un-fat Hadoop from Flink fat jar This PR implements FLINK-5998 . It marks all Hadoop dependency in the dist jar as `provided` so that users can plug in the jars from their own Hadoop distribution. You can merge this pull request into a Git repository by running: $ git pull https://github.com/haohui/flink FLINK-5998 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3604.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3604 commit a8f145cce141450edeb92c5ea9bed6d0441dc94f Author: Haohui Mai <wheat9@apache.org> Date: 2017-03-23T21:46:45Z FLINK-5998 Un-fat Hadoop from Flink fat jar.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3604

          I have tested this change locally, but it does not exactly what my intention was.

          What I want is that the `lib/` folder contains a `flink-dist.jar` with all the Flink runtime dependencies, and a `flink-dist-hadoop.jar` with the packaged Hadoop dependencies.
          This way, users have a good out of the box experience, but they can use their own Hadoop jars easily.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3604 I have tested this change locally, but it does not exactly what my intention was. What I want is that the `lib/` folder contains a `flink-dist.jar` with all the Flink runtime dependencies, and a `flink-dist-hadoop.jar` with the packaged Hadoop dependencies. This way, users have a good out of the box experience, but they can use their own Hadoop jars easily.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3604

          @rmetzger, how would users know which Hadoop dependencies to include in `flink-dist-hadoop.jar`? Would they be copying multiple component jars into `lib/`?

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3604 @rmetzger, how would users know which Hadoop dependencies to include in `flink-dist-hadoop.jar`? Would they be copying multiple component jars into `lib/`?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3604

          Yes, the idea is that users in some environments can even delete `flink-dist-hadoop.jar` entirely, and just configure the classpath correctly to point to the Hadoop lib folder of their hadoop distribution.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3604 Yes, the idea is that users in some environments can even delete `flink-dist-hadoop.jar` entirely, and just configure the classpath correctly to point to the Hadoop lib folder of their hadoop distribution.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3604

          @rmetzger thanks for the clarification. Sounds good!

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3604 @rmetzger thanks for the clarification. Sounds good!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user haohui commented on the issue:

          https://github.com/apache/flink/pull/3604

          @rmetzger – thanks for the clarification. Do you think that it is sufficient to mark the dependency of `flink-shaded-hadoop2` as provided and produces a dedicated jar for it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user haohui commented on the issue: https://github.com/apache/flink/pull/3604 @rmetzger – thanks for the clarification. Do you think that it is sufficient to mark the dependency of `flink-shaded-hadoop2` as provided and produces a dedicated jar for it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3604

          Mh, not sure.
          What I would try is updating the assembly descriptor(s) in the flink-dist project, so that it creates two assemblies.
          Afaik you need to define all dependencies as regular deps in flink-dist, and then include subsets of them into different assemblies.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3604 Mh, not sure. What I would try is updating the assembly descriptor(s) in the flink-dist project, so that it creates two assemblies. Afaik you need to define all dependencies as regular deps in flink-dist, and then include subsets of them into different assemblies.
          Hide
          wheat9 Haohui Mai added a comment -

          All failed tests passed locally. :-|

          Robert Metzger, is it possible for you can take a look? Much appreciated.

          Show
          wheat9 Haohui Mai added a comment - All failed tests passed locally. :-| Robert Metzger , is it possible for you can take a look? Much appreciated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user haohui commented on the issue:

          https://github.com/apache/flink/pull/3604

          @rmetzger ping...would you mind taking a look?

          Show
          githubbot ASF GitHub Bot added a comment - Github user haohui commented on the issue: https://github.com/apache/flink/pull/3604 @rmetzger ping...would you mind taking a look?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3604

          @haohui I think this goes into the right direction, but the `flink-shaded-hadoop2-1.3.jar` should be in `/lib`, not in `opt`. That way, we preserve the original behavior (Hadoop dependencies are available by default), but make it easy to swap/remove the Hadoop dependency.

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3604 @haohui I think this goes into the right direction, but the `flink-shaded-hadoop2-1.3.jar` should be in `/lib`, not in `opt`. That way, we preserve the original behavior (Hadoop dependencies are available by default), but make it easy to swap/remove the Hadoop dependency.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3604

          Sorry for the delay. I'm now checking out the change....

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3604 Sorry for the delay. I'm now checking out the change....
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3604

          Success!

          I got it to work on my CDH 5.4.2 virtual machine with the following environment variables set
          ```
          export HADOOP_CLASSPATH=/usr/lib/hadoop-yarn/:/usr/lib/hadoop/:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop/lib/:/etc/hadoop/conf:/usr/lib/hadoop-hdfs/*
          ```

          and it was using CDH's Hadoop version:
          ```
          2017-05-04 10:49:58,243 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Hadoop version: 2.6.0-cdh5.4.2
          ```

          I'll merge this change to master.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3604 Success! I got it to work on my CDH 5.4.2 virtual machine with the following environment variables set ``` export HADOOP_CLASSPATH=/usr/lib/hadoop-yarn/ :/usr/lib/hadoop/ :/usr/lib/hadoop-yarn/lib/ :/usr/lib/hadoop/lib/ :/etc/hadoop/conf:/usr/lib/hadoop-hdfs/* ``` and it was using CDH's Hadoop version: ``` 2017-05-04 10:49:58,243 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Hadoop version: 2.6.0-cdh5.4.2 ``` I'll merge this change to master.
          Show
          rmetzger Robert Metzger added a comment - Merged in http://git-wip-us.apache.org/repos/asf/flink/commit/43fa5071
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3604

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3604
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user aljoscha opened a pull request:

          https://github.com/apache/flink/pull/3856

          FLINK-5998 Fix shaded Hadoop Jar

          Now that flink-dist has the shaded hadoop jar as a provided dependency
          we need to ensure that the shaded hadoop jar includes all the
          dependencies that it needs because flink-dist does not include them
          anymore in its shading.

          The manifestation of this was that we couldn't start a cluster anymore
          because of missing class files.

          The upside of this PR is that we can run a cluster again. The downside is that we now have some dependencies both in the `flink-dist` jar and in the shaded Hadoop jar, i.e. total size increases.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/aljoscha/flink fix-shaded-hadoop

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3856.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3856


          commit ce0eb550277b9dd737f4dabb4eb586d36bd6ed35
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-05-09T13:50:09Z

          FLINK-5998 Fix shaded Hadoop Jar

          Now that flink-dist has the shaded hadoop jar as a provided dependency
          we need to ensure that the shaded hadoop jar includes all the
          dependencies that it needs because flink-dist does not include them
          anymore in its shading.

          The manifestation of this was that we couldn't start a cluster anymore
          because of missing class files.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/3856 FLINK-5998 Fix shaded Hadoop Jar Now that flink-dist has the shaded hadoop jar as a provided dependency we need to ensure that the shaded hadoop jar includes all the dependencies that it needs because flink-dist does not include them anymore in its shading. The manifestation of this was that we couldn't start a cluster anymore because of missing class files. The upside of this PR is that we can run a cluster again. The downside is that we now have some dependencies both in the `flink-dist` jar and in the shaded Hadoop jar, i.e. total size increases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink fix-shaded-hadoop Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3856.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3856 commit ce0eb550277b9dd737f4dabb4eb586d36bd6ed35 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-05-09T13:50:09Z FLINK-5998 Fix shaded Hadoop Jar Now that flink-dist has the shaded hadoop jar as a provided dependency we need to ensure that the shaded hadoop jar includes all the dependencies that it needs because flink-dist does not include them anymore in its shading. The manifestation of this was that we couldn't start a cluster anymore because of missing class files.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/3856

          R: @rmetzger Could you please have a quick look, if this makes sense?

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3856 R: @rmetzger Could you please have a quick look, if this makes sense?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user gyfora commented on the issue:

          https://github.com/apache/flink/pull/3856

          This fix seems to work for me as well 👍

          Show
          githubbot ASF GitHub Bot added a comment - Github user gyfora commented on the issue: https://github.com/apache/flink/pull/3856 This fix seems to work for me as well 👍
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3856

          I think the fix works in almost all cases.
          There's only one problem: Your change causes the `flink-shaded-hadoop2` artifact on maven central to not expose any dependencies anymore. So a module referencing this (like `flink-java`) will not see what `flink-shaded-hadoop2` contains. This can cause problems like having classes in the classpath multiple times. Maven can not "manage" the dependencies in that case, because it does not know what's `flink-shaded-hadoop2`.

          I don't have a good answer how to solve this.
          Some ideas:

          • relocate all Hadoop dependencies in the `flink-shaded-hadoop2` artifact. Then, we won't run into the original problem anymore. I've tried this once, but I was running into problems getting the YARN tests running afterwards
          • introduce a special "flink-shaded-hadoop2-dist" module that prepares a fat dist hadoop jar for the binary distribution. This way, we can differentiate between `flink-shaded-hadoop2` as a dependency and for the binary. But I think this will lead to problems when building `flink-dist`...
          • merge this PR as is and hope that the problems don't occur (I think this is mostly relevant for people using Hadoop code in their user jar, for example when doing some Hadoop compatibility stuff)
          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3856 I think the fix works in almost all cases. There's only one problem: Your change causes the `flink-shaded-hadoop2` artifact on maven central to not expose any dependencies anymore. So a module referencing this (like `flink-java`) will not see what `flink-shaded-hadoop2` contains. This can cause problems like having classes in the classpath multiple times. Maven can not "manage" the dependencies in that case, because it does not know what's `flink-shaded-hadoop2`. I don't have a good answer how to solve this. Some ideas: relocate all Hadoop dependencies in the `flink-shaded-hadoop2` artifact. Then, we won't run into the original problem anymore. I've tried this once, but I was running into problems getting the YARN tests running afterwards introduce a special "flink-shaded-hadoop2-dist" module that prepares a fat dist hadoop jar for the binary distribution. This way, we can differentiate between `flink-shaded-hadoop2` as a dependency and for the binary. But I think this will lead to problems when building `flink-dist`... merge this PR as is and hope that the problems don't occur (I think this is mostly relevant for people using Hadoop code in their user jar, for example when doing some Hadoop compatibility stuff)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/3856

          @gyfora See https://issues.apache.org/jira/browse/FLINK-6514, it seems that this only occurs with newer Maven versions. When building with 3.2.5 it works.

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3856 @gyfora See https://issues.apache.org/jira/browse/FLINK-6514 , it seems that this only occurs with newer Maven versions. When building with 3.2.5 it works.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user gyfora commented on the issue:

          https://github.com/apache/flink/pull/3856

          @aljoscha Yes, that might work although I prefer this fix compared to having to mess with the maven versions on deploy servers.

          Show
          githubbot ASF GitHub Bot added a comment - Github user gyfora commented on the issue: https://github.com/apache/flink/pull/3856 @aljoscha Yes, that might work although I prefer this fix compared to having to mess with the maven versions on deploy servers.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3856

          I think Stephan is working on fixing this properly for all Maven versions.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3856 I think Stephan is working on fixing this properly for all Maven versions.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3856

          Afaik this PR can be closed because it has been superseded by Stephan's PR.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3856 Afaik this PR can be closed because it has been superseded by Stephan's PR.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/3856

          Superseded

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3856 Superseded
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha closed the pull request at:

          https://github.com/apache/flink/pull/3856

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/3856

            People

            • Assignee:
              wheat9 Haohui Mai
              Reporter:
              rmetzger Robert Metzger
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development