Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2954

update shade configurations in extension/sql

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: dsl-sql
    • Labels:

      Description

      guava is shaded by default in beam modules, while calcite(a dependency of SQL) includes guava which is not shaded. Below error is thrown when calling calcite methods with guava classed.

      Exception in thread "main" java.lang.NoSuchMethodError: org.apache.calcite.rel.core.Aggregate.getGroupSets()Lorg/apache/beam/sdks/java/extensions/sql/repackaged/com/google/common/collect/ImmutableList;
      	at org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.updateWindowTrigger(BeamAggregationRule.java:139)
      	at org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.onMatch(BeamAggregationRule.java:73)
      	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
      	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
      	at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
      	at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:313)
      	at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:149)
      	at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateAndConvert(BeamQueryPlanner.java:140)
      	at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:128)
      	at org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.compileBeamPipeline(BeamQueryPlanner.java:113)
      	at org.apache.beam.sdk.extensions.sql.impl.BeamSqlCli.compilePipeline(BeamSqlCli.java:62)
      

        Issue Links

          Activity

          Hide
          aviemzur Aviem Zur added a comment - - edited

          We cannot opt-out of the shading rules in the SQL module as we cannot leak Guava to our users.

          If a shaded version of Calcite which relocates Guava exists we should use that (such a version does not seem to exist). Otherwise, since Calcite is under Apache License 2.0 we can shade it into our SQL module, using the same relocation rules applied in beam-parent.

          Show
          aviemzur Aviem Zur added a comment - - edited We cannot opt-out of the shading rules in the SQL module as we cannot leak Guava to our users. If a shaded version of Calcite which relocates Guava exists we should use that (such a version does not seem to exist). Otherwise, since Calcite is under Apache License 2.0 we can shade it into our SQL module, using the same relocation rules applied in beam-parent .
          Hide
          mingmxu Xu Mingmin added a comment -

          make sense to me, I'll turn to shade Calcite dependencies.

          Show
          mingmxu Xu Mingmin added a comment - make sense to me, I'll turn to shade Calcite dependencies.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user XuMingmin opened a pull request:

          https://github.com/apache/beam/pull/3848

          BEAM-2954 update shade configurations in extension/sql

          Follow this checklist to help us incorporate your contribution quickly and easily:

          • [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
          • [ ] Each commit in the pull request should have a meaningful subject line and body.
          • [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
          • [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
          • [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
          • [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/XuMingmin/beam BEAM-2954

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3848.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3848


          commit 522e6c188809327f098b84dbd7ee07e1dad4a376
          Author: mingmxu <mingmxu@ebay.com>
          Date: 2017-09-13T17:31:19Z

          update shade settings to handle Calcite dependencies


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user XuMingmin opened a pull request: https://github.com/apache/beam/pull/3848 BEAM-2954 update shade configurations in extension/sql Follow this checklist to help us incorporate your contribution quickly and easily: [ ] Make sure there is a [JIRA issue] ( https://issues.apache.org/jira/projects/BEAM/issues/ ) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. [ ] Each commit in the pull request should have a meaningful subject line and body. [ ] Format the pull request title like ` [BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/XuMingmin/beam BEAM-2954 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3848.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3848 commit 522e6c188809327f098b84dbd7ee07e1dad4a376 Author: mingmxu <mingmxu@ebay.com> Date: 2017-09-13T17:31:19Z update shade settings to handle Calcite dependencies

            People

            • Assignee:
              mingmxu Xu Mingmin
              Reporter:
              mingmxu Xu Mingmin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development