Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6654

missing maven dependency on "flink-shaded-hadoop2-uber" in flink-dist

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.4.0, 1.3.2
    • Component/s: Build System
    • Labels:
      None

      Description

      Since applying FLINK-6514, flink-dist includes flink-shaded-hadoop2-uber-*.jar but without giving this dependency in its pom.xml. This may lead to concurrency issues during builds but also fails building the flink-dist module only (with dependencies) as in

      mvn clean install -pl flink-dist -am
      

        Issue Links

          Activity

          Hide
          aljoscha Aljoscha Krettek added a comment -

          Fixed in

          • 1.3.2 via 01265fe1529b6712311051ed2d210b5e2e5cf0a8
          • 1.4.0 via 70d53487a7f619d97aaad49879ad4cf7f5704adb
          Show
          aljoscha Aljoscha Krettek added a comment - Fixed in 1.3.2 via 01265fe1529b6712311051ed2d210b5e2e5cf0a8 1.4.0 via 70d53487a7f619d97aaad49879ad4cf7f5704adb
          Hide
          aljoscha Aljoscha Krettek added a comment -

          Reopen to fix release note.

          Show
          aljoscha Aljoscha Krettek added a comment - Reopen to fix release note.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3960

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3960
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3960

          I think this looks correct now. Merging for `master` and `release-1.3`...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3960 I think this looks correct now. Merging for `master` and `release-1.3`...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3960

          I'm not sure if the proposed fix is correct, because it will add the shaded hadoop uber project's dependencies into Flink dist.
          IIRC, this was the whole point of Stephan's recent refactoring of the dependencies.

          I did a quick check, just based on the flink-dist file size and content, and with the current rel 1.3 branch, we have the following file sizes:
          ```
          rw-rr- 1 robert robert 68M May 24 09:47 flink-dist_2.10-1.3-SNAPSHOT.jar
          rw-rr- 1 robert robert 99K May 24 09:40 flink-python_2.10-1.3-SNAPSHOT.jar
          rw-rr- 1 robert robert 23M May 24 09:47 flink-shaded-hadoop2-uber-1.3-SNAPSHOT.jar
          rw-rr- 1 robert robert 479K Nov 17 2015 log4j-1.2.17.jar
          rw-rr- 1 robert robert 8.7K Nov 17 2015 slf4j-log4j12-1.7.7.jar
          ```

          after the change, the flink dist file grows significantly:
          ```
          rw-rr- 1 robert robert 87M May 24 09:51 flink-dist_2.10-1.3-SNAPSHOT.jar
          rw-rr- 1 robert robert 99K May 24 09:40 flink-python_2.10-1.3-SNAPSHOT.jar
          rw-rr- 1 robert robert 23M May 24 09:47 flink-shaded-hadoop2-uber-1.3-SNAPSHOT.jar
          rw-rr- 1 robert robert 479K Nov 17 2015 log4j-1.2.17.jar
          rw-rr- 1 robert robert 8.7K Nov 17 2015 slf4j-log4j12-1.7.7.jar
          ```
          I quickly opened the file and I found all `org.apache.hadoop` classes in the dist again.

          I'm against doing a "quick fix" before the release here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3960 I'm not sure if the proposed fix is correct, because it will add the shaded hadoop uber project's dependencies into Flink dist. IIRC, this was the whole point of Stephan's recent refactoring of the dependencies. I did a quick check, just based on the flink-dist file size and content, and with the current rel 1.3 branch, we have the following file sizes: ``` rw-r r - 1 robert robert 68M May 24 09:47 flink-dist_2.10-1.3-SNAPSHOT.jar rw-r r - 1 robert robert 99K May 24 09:40 flink-python_2.10-1.3-SNAPSHOT.jar rw-r r - 1 robert robert 23M May 24 09:47 flink-shaded-hadoop2-uber-1.3-SNAPSHOT.jar rw-r r - 1 robert robert 479K Nov 17 2015 log4j-1.2.17.jar rw-r r - 1 robert robert 8.7K Nov 17 2015 slf4j-log4j12-1.7.7.jar ``` after the change, the flink dist file grows significantly: ``` rw-r r - 1 robert robert 87M May 24 09:51 flink-dist_2.10-1.3-SNAPSHOT.jar rw-r r - 1 robert robert 99K May 24 09:40 flink-python_2.10-1.3-SNAPSHOT.jar rw-r r - 1 robert robert 23M May 24 09:47 flink-shaded-hadoop2-uber-1.3-SNAPSHOT.jar rw-r r - 1 robert robert 479K Nov 17 2015 log4j-1.2.17.jar rw-r r - 1 robert robert 8.7K Nov 17 2015 slf4j-log4j12-1.7.7.jar ``` I quickly opened the file and I found all `org.apache.hadoop` classes in the dist again. I'm against doing a "quick fix" before the release here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3960

          Okay, I was able to reproduce the issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3960 Okay, I was able to reproduce the issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user NicoK commented on the issue:

          https://github.com/apache/flink/pull/3960

          you probably need to clean up your whole source with `mvn clean` and try from there - I'm not quite sure, my initial command will clean up everything that is not in the dependencies of the flink-dist project

          Show
          githubbot ASF GitHub Bot added a comment - Github user NicoK commented on the issue: https://github.com/apache/flink/pull/3960 you probably need to clean up your whole source with `mvn clean` and try from there - I'm not quite sure, my initial command will clean up everything that is not in the dependencies of the flink-dist project
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3960

          Not sure about the initial issue. For me the command worked (without yoru changes on the 1.3 branch)

          ```
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 10:12 min
          [INFO] Finished at: 2017-05-23T16:50:31+02:00
          [INFO] Final Memory: 131M/441M
          [INFO] ------------------------------------------------------------------------
          mvn clean install -pl flink-dist -am -DskipTests
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3960 Not sure about the initial issue. For me the command worked (without yoru changes on the 1.3 branch) ``` [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10:12 min [INFO] Finished at: 2017-05-23T16:50:31+02:00 [INFO] Final Memory: 131M/441M [INFO] ------------------------------------------------------------------------ mvn clean install -pl flink-dist -am -DskipTests ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user NicoK commented on the issue:

          https://github.com/apache/flink/pull/3960

          I wasn't quite sure and went the safe way - can remove the exclusion though...

          Show
          githubbot ASF GitHub Bot added a comment - Github user NicoK commented on the issue: https://github.com/apache/flink/pull/3960 I wasn't quite sure and went the safe way - can remove the exclusion though...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3960

          Thanks. The exclusion should not be necessary, since the `flink-shaded-hadoop2-uber` dependency shades everything into one jar and has no further dependencies.

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3960 Thanks. The exclusion should not be necessary, since the `flink-shaded-hadoop2-uber` dependency shades everything into one jar and has no further dependencies.
          Hide
          aljoscha Aljoscha Krettek added a comment -

          I think the dependency is not included on purpose. Stephan Ewen, could you please chime in?

          Show
          aljoscha Aljoscha Krettek added a comment - I think the dependency is not included on purpose. Stephan Ewen , could you please chime in?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user NicoK opened a pull request:

          https://github.com/apache/flink/pull/3960

          FLINK-6654[build] let 'flink-dist' properly depend on 'flink-shaded…

          Since applying FLINK-6514, flink-dist includes `flink-shaded-hadoop2-uber-*.jar` but without giving this dependency in its `pom.xml`. This may lead to concurrency issues during builds but also fails building the flink-dist module only (with dependencies) as in

          `mvn clean install -pl flink-dist -am`

          This PR adds the missing dependency.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/NicoK/flink flink-6654

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3960.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3960


          commit 1b40449ac5373e93f8ddb4b3cf46dca2bcc3a900
          Author: Nico Kruber <nico@data-artisans.com>
          Date: 2017-05-22T10:14:59Z

          FLINK-6654[build] let 'flink-dist' properly depend on 'flink-shaded-hadoop2-uber'


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/3960 FLINK-6654 [build] let 'flink-dist' properly depend on 'flink-shaded… Since applying FLINK-6514 , flink-dist includes `flink-shaded-hadoop2-uber-*.jar` but without giving this dependency in its `pom.xml`. This may lead to concurrency issues during builds but also fails building the flink-dist module only (with dependencies) as in `mvn clean install -pl flink-dist -am` This PR adds the missing dependency. You can merge this pull request into a Git repository by running: $ git pull https://github.com/NicoK/flink flink-6654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3960.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3960 commit 1b40449ac5373e93f8ddb4b3cf46dca2bcc3a900 Author: Nico Kruber <nico@data-artisans.com> Date: 2017-05-22T10:14:59Z FLINK-6654 [build] let 'flink-dist' properly depend on 'flink-shaded-hadoop2-uber'
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user NicoK closed the pull request at:

          https://github.com/apache/flink/pull/3959

          Show
          githubbot ASF GitHub Bot added a comment - Github user NicoK closed the pull request at: https://github.com/apache/flink/pull/3959
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user NicoK commented on the issue:

          https://github.com/apache/flink/pull/3959

          sorry, wrong branch to submit to

          Show
          githubbot ASF GitHub Bot added a comment - Github user NicoK commented on the issue: https://github.com/apache/flink/pull/3959 sorry, wrong branch to submit to
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user NicoK opened a pull request:

          https://github.com/apache/flink/pull/3959

          FLINK-6654 let 'flink-dist' properly depend on 'flink-shaded-hadoop2-uber'

          Since applying FLINK-6514, flink-dist includes `flink-shaded-hadoop2-uber-*.jar` but without giving this dependency in its `pom.xml`. This may lead to concurrency issues during builds but also fails building the flink-dist module only (with dependencies) as in

          `mvn clean install -pl flink-dist -am`

          This PR adds the missing dependency.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/NicoK/flink flink-6654

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3959.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3959



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/3959 FLINK-6654 let 'flink-dist' properly depend on 'flink-shaded-hadoop2-uber' Since applying FLINK-6514 , flink-dist includes `flink-shaded-hadoop2-uber-*.jar` but without giving this dependency in its `pom.xml`. This may lead to concurrency issues during builds but also fails building the flink-dist module only (with dependencies) as in `mvn clean install -pl flink-dist -am` This PR adds the missing dependency. You can merge this pull request into a Git repository by running: $ git pull https://github.com/NicoK/flink flink-6654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3959.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3959

            People

            • Assignee:
              NicoK Nico Kruber
              Reporter:
              NicoK Nico Kruber
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development