Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6247

Build a jar-with-dependencies for flink-table and put it into ./opt

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.0
    • Labels:
      None

      Description

      Due to a problem with Calcite and the unloading of classes, user-code classloaders that include Calcite cannot be garbage collected. This is a problem for long-running clusters that execute multiple Table API / SQL programs with fat JARs that include the flink-table dependency. Each executed program comes with an own user-code classloader that cannot be cleaned up later.

      As a workaround, we recommend to copy the flink-table dependency into the ./lib folder. However, we do not have a jar file with all required transitive dependencies (Calcite, Janino, etc). Hence, users would need to build this jar file themselves or copy all jars into ./lib.

      This issue is about creating a jar-with-dependencies and adding it to the ./opt folder. Users can then copy the jar file from ./opt to ./lib to include the table API in the classpath of Flink.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sunjincheng121 opened a pull request:

          https://github.com/apache/flink/pull/3666

          FLINK-6247[table] Build a jar-with-dependencies for flink-table and…

          … put it into ./opt

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [x] General
          • The pull request references the related JIRA issue ("FLINK-6247 Build a jar-with-dependencies for flink-table and put it into ./opt")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [ ] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [ ] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/sunjincheng121/flink FLINK-6247-PR

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3666.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3666


          commit 5c2ddf580d23e7745499c0d5f639b18d766e6d24
          Author: sunjincheng121 <sunjincheng121@gmail.com>
          Date: 2017-03-31T08:31:18Z

          FLINK-6247[table] Build a jar-with-dependencies for flink-table and put it into ./opt


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sunjincheng121 opened a pull request: https://github.com/apache/flink/pull/3666 FLINK-6247 [table] Build a jar-with-dependencies for flink-table and… … put it into ./opt Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [x] General The pull request references the related JIRA issue (" FLINK-6247 Build a jar-with-dependencies for flink-table and put it into ./opt") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [ ] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [ ] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/sunjincheng121/flink FLINK-6247 -PR Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3666.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3666 commit 5c2ddf580d23e7745499c0d5f639b18d766e6d24 Author: sunjincheng121 <sunjincheng121@gmail.com> Date: 2017-03-31T08:31:18Z FLINK-6247 [table] Build a jar-with-dependencies for flink-table and put it into ./opt
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3666#discussion_r109594633

          — Diff: flink-libraries/flink-table/pom.xml —
          @@ -40,6 +40,12 @@ under the License.
          <groupId>org.apache.flink</groupId>
          <artifactId>flink-streaming-scala_2.10</artifactId>
          <version>$

          {project.version}

          </version>
          + <exclusions>
          — End diff –

          We should set this dependency to `<scope>provided</scope>` instead of excluding all of its dependencies.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3666#discussion_r109594633 — Diff: flink-libraries/flink-table/pom.xml — @@ -40,6 +40,12 @@ under the License. <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-scala_2.10</artifactId> <version>$ {project.version} </version> + <exclusions> — End diff – We should set this dependency to `<scope>provided</scope>` instead of excluding all of its dependencies.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3666#discussion_r112152700

          — Diff: flink-dist/src/main/assemblies/opt.xml —
          @@ -66,6 +66,14 @@
          <fileMode>0644</fileMode>
          </file>

          + <!-- TableAPI-->
          + <file>
          + <source>../flink-libraries/flink-table/target/flink-table_2.10-$

          {project.version}-jar-with-dependencies.jar</source>
          — End diff –

          copy the default jar instead of the `jar-with-dependencies`, i.e., `../flink-libraries/flink-table/target/flink-table_2.10-${project.version}

          .jar`

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3666#discussion_r112152700 — Diff: flink-dist/src/main/assemblies/opt.xml — @@ -66,6 +66,14 @@ <fileMode>0644</fileMode> </file> + <!-- TableAPI--> + <file> + <source>../flink-libraries/flink-table/target/flink-table_2.10-$ {project.version}-jar-with-dependencies.jar</source> — End diff – copy the default jar instead of the `jar-with-dependencies`, i.e., `../flink-libraries/flink-table/target/flink-table_2.10-${project.version} .jar`
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3666#discussion_r112152396

          — Diff: flink-libraries/flink-table/pom.xml —
          @@ -231,6 +237,25 @@ under the License.
          </configuration>
          </plugin>

          + <!-- build a jar-with-dependencies, to be included in the 'opt' build folder -->
          + <plugin>
          — End diff –

          Let's build the jar with dependencies as default jar using the shade plugin.
          Currently, the default jar already contains most dependencies, including Calcite (with relocates Guava) and Reflections. We only need to add Janino to the shade plugin configuration (add `<include>org.codehaus.janino:</include>` after `<include>org.reflections:</include>`) and then we can use the default jar and put it into `./opt` because it contains all dependencies.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3666#discussion_r112152396 — Diff: flink-libraries/flink-table/pom.xml — @@ -231,6 +237,25 @@ under the License. </configuration> </plugin> + <!-- build a jar-with-dependencies, to be included in the 'opt' build folder --> + <plugin> — End diff – Let's build the jar with dependencies as default jar using the shade plugin. Currently, the default jar already contains most dependencies, including Calcite (with relocates Guava) and Reflections. We only need to add Janino to the shade plugin configuration (add `<include>org.codehaus.janino: </include>` after `<include>org.reflections: </include>`) and then we can use the default jar and put it into `./opt` because it contains all dependencies.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3666#discussion_r112201348

          — Diff: flink-libraries/flink-table/pom.xml —
          @@ -40,6 +40,12 @@ under the License.
          <groupId>org.apache.flink</groupId>
          <artifactId>flink-streaming-scala_2.10</artifactId>
          <version>$

          {project.version}

          </version>
          + <exclusions>
          — End diff –

          I noticed that this change causes the build to fail because `flink-hbase` and `flink-connector-kafka-0.10` depend on `flink-table` to pull required dependencies. The missing dependencies need to be added to their pom files to fix the build.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3666#discussion_r112201348 — Diff: flink-libraries/flink-table/pom.xml — @@ -40,6 +40,12 @@ under the License. <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-scala_2.10</artifactId> <version>$ {project.version} </version> + <exclusions> — End diff – I noticed that this change causes the build to fail because `flink-hbase` and `flink-connector-kafka-0.10` depend on `flink-table` to pull required dependencies. The missing dependencies need to be added to their pom files to fix the build.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sunjincheng121 commented on the issue:

          https://github.com/apache/flink/pull/3666

          Hi @fhueske Thanks for your Reviewing. And sorry for late reply. I have updated the PR. Please take a look. Thanks a lot.

          Thanks,
          SunJincheng

          Show
          githubbot ASF GitHub Bot added a comment - Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3666 Hi @fhueske Thanks for your Reviewing. And sorry for late reply. I have updated the PR. Please take a look. Thanks a lot. Thanks, SunJincheng
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sunjincheng121 commented on the issue:

          https://github.com/apache/flink/pull/3666

          Hi @fhueske Thanks for your Reviewing. And sorry for late reply. I have updated the PR. Please take a look. Thanks a lot.

          Thanks,
          SunJincheng

          Show
          githubbot ASF GitHub Bot added a comment - Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3666 Hi @fhueske Thanks for your Reviewing. And sorry for late reply. I have updated the PR. Please take a look. Thanks a lot. Thanks, SunJincheng
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on the issue:

          https://github.com/apache/flink/pull/3666

          Thanks for the update.
          Validated that the `flink-table` jar file in `./opt/` comes with all dependencies to run a Table API job packaged in a jar without any additional classes.

          Will merge this PR

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3666 Thanks for the update. Validated that the `flink-table` jar file in `./opt/` comes with all dependencies to run a Table API job packaged in a jar without any additional classes. Will merge this PR
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3666

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3666
          Hide
          fhueske Fabian Hueske added a comment -

          Fixed for 1.3 with 69905e19c636a05992443a647ba4142af5a40a01

          Show
          fhueske Fabian Hueske added a comment - Fixed for 1.3 with 69905e19c636a05992443a647ba4142af5a40a01
          Hide
          xccui Xingcan Cui added a comment -

          Sorry for reawakening this. I know setting the dependencies with "provided" for flink-examples-table is to avoid conflict for distribution. But that makes the examples not-executable from IDE, which seems to be a little inconvenience. Shall we find a better way to solve this or just let it be?

          Show
          xccui Xingcan Cui added a comment - Sorry for reawakening this. I know setting the dependencies with "provided" for flink-examples-table is to avoid conflict for distribution. But that makes the examples not-executable from IDE, which seems to be a little inconvenience. Shall we find a better way to solve this or just let it be?
          Hide
          fhueske Fabian Hueske added a comment -

          Hi Xingcan Cui, thanks for reporting this. I agree, the examples should be executable from the IDE.
          I think we could fix this by changing the Flink dependencies in flink-examples-table to not provided.

          What do you think?

          Show
          fhueske Fabian Hueske added a comment - Hi Xingcan Cui , thanks for reporting this. I agree, the examples should be executable from the IDE. I think we could fix this by changing the Flink dependencies in flink-examples-table to not provided. What do you think?
          Hide
          xccui Xingcan Cui added a comment -

          Hi Fabian Hueske, as long as it doesn't matter for the target jar, I think this should be the most direct and effective solution.

          Show
          xccui Xingcan Cui added a comment - Hi Fabian Hueske , as long as it doesn't matter for the target jar, I think this should be the most direct and effective solution.

            People

            • Assignee:
              sunjincheng121 sunjincheng
              Reporter:
              fhueske Fabian Hueske
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development