Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Table API & SQL
    • Labels:
      None

      Description

      some UDAGGs have multi-fields as input. For instance,
      table
      .window(Tumble over 10.minutes on 'rowtime as 'w )
      .groupBy('key, 'w)
      .select('key, weightedAvg('value, 'weight))

      This task will add the support for the aggregate on multi fields.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user shaoxuan-wang opened a pull request:

          https://github.com/apache/flink/pull/3647

          FLINK-5915 [table] forward the entire aggregate ArgList to aggregate runtime functions

          This PR partially solved "FLINK-5915 Add support for the aggregate on multi fields".
          The roadmap of UDAGG would be 1. codeGen all the runtime aggregate functions; 2. add UDAGG tableAPI interface.
          I would like to kick this PR off earlier as: a) we can finalize the runtime function while doing the codeGen; b) as more and more (over) aggregates are implemented, it would be good if we can finalize the interface and share library to the stage (as we planned) as earlier as possible.

          Note that: though the entire aggregate ArgList is forwarded to the runtime function, for the functions that have not been codeGened, we will still support only one column aggregate input.

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [ ] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [ ] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [ ] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/shaoxuan-wang/flink FLINK5915-submit

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3647.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3647


          commit 128479c24003e8f0bd67542f336f9ae20dade352
          Author: shaoxuan-wang <wshaoxuan@gmail.com>
          Date: 2017-03-29T20:41:38Z

          FLINK-5915 [table] forward the entire aggregate ArgList to aggregate runtime functions


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user shaoxuan-wang opened a pull request: https://github.com/apache/flink/pull/3647 FLINK-5915 [table] forward the entire aggregate ArgList to aggregate runtime functions This PR partially solved " FLINK-5915 Add support for the aggregate on multi fields". The roadmap of UDAGG would be 1. codeGen all the runtime aggregate functions; 2. add UDAGG tableAPI interface. I would like to kick this PR off earlier as: a) we can finalize the runtime function while doing the codeGen; b) as more and more (over) aggregates are implemented, it would be good if we can finalize the interface and share library to the stage (as we planned) as earlier as possible. Note that: though the entire aggregate ArgList is forwarded to the runtime function, for the functions that have not been codeGened, we will still support only one column aggregate input. Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [ ] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [ ] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [ ] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/shaoxuan-wang/flink FLINK5915-submit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3647.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3647 commit 128479c24003e8f0bd67542f336f9ae20dade352 Author: shaoxuan-wang <wshaoxuan@gmail.com> Date: 2017-03-29T20:41:38Z FLINK-5915 [table] forward the entire aggregate ArgList to aggregate runtime functions
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3647#discussion_r109032005

          — Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateUtil.scala —
          @@ -857,28 +857,30 @@ object AggregateUtil {
          aggregateCalls: Seq[AggregateCall],
          inputType: RelDataType,
          needRetraction: Boolean)

          • : (Array[Int], Array[TableAggregateFunction[_ <: Any]]) = {
            + : (Array[util.List[Integer]], Array[TableAggregateFunction[_ <: Any]]) = {
              • End diff –

          Can we use an `Array[Int]` instead of a `List[Integer]`?

          Scala `Int` is compiled to Java `int` primitive. `java.lang.Integer` is a boxed type.
          Also an array access is faster than a `List.get()` access (`ArrayList.get()` would be a bit better).

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/3647#discussion_r109032005 — Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateUtil.scala — @@ -857,28 +857,30 @@ object AggregateUtil { aggregateCalls: Seq [AggregateCall] , inputType: RelDataType, needRetraction: Boolean) : (Array [Int] , Array[TableAggregateFunction [_ <: Any] ]) = { + : (Array[util.List [Integer] ], Array[TableAggregateFunction [_ <: Any] ]) = { End diff – Can we use an `Array [Int] ` instead of a `List [Integer] `? Scala `Int` is compiled to Java `int` primitive. `java.lang.Integer` is a boxed type. Also an array access is faster than a `List.get()` access (`ArrayList.get()` would be a bit better).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on the issue:

          https://github.com/apache/flink/pull/3647

          The change looks good in general. I'd prefer to use an array instead of a List though.
          What do you think @shaoxuan-wang?

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3647 The change looks good in general. I'd prefer to use an array instead of a List though. What do you think @shaoxuan-wang?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user shaoxuan-wang commented on the issue:

          https://github.com/apache/flink/pull/3647

          Thanks @fhueske, I overlooked a list is not always an arraylist. I change it to scala array, but keep the type as integer, as this is type returned (and not easy to be casted to Int) from aggregateCall.getArgList. I updated PR and also rebased to the master (as I noticed there are a few over aggregates have been recently merged), so it will be easier for you to merge.

          Show
          githubbot ASF GitHub Bot added a comment - Github user shaoxuan-wang commented on the issue: https://github.com/apache/flink/pull/3647 Thanks @fhueske, I overlooked a list is not always an arraylist. I change it to scala array, but keep the type as integer, as this is type returned (and not easy to be casted to Int) from aggregateCall.getArgList. I updated PR and also rebased to the master (as I noticed there are a few over aggregates have been recently merged), so it will be easier for you to merge.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fhueske commented on the issue:

          https://github.com/apache/flink/pull/3647

          Thanks @shaoxuan-wang, I found a way to cast the `Integer` to `Int`. Will change that and merge the PR.

          Thanks, Fabian

          Show
          githubbot ASF GitHub Bot added a comment - Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3647 Thanks @shaoxuan-wang, I found a way to cast the `Integer` to `Int`. Will change that and merge the PR. Thanks, Fabian
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3647

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3647
          Hide
          ShaoxuanWang Shaoxuan Wang added a comment -

          This is completely resolved with FLINK-5906.

          Show
          ShaoxuanWang Shaoxuan Wang added a comment - This is completely resolved with FLINK-5906 .

            People

            • Assignee:
              ShaoxuanWang Shaoxuan Wang
              Reporter:
              ShaoxuanWang Shaoxuan Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development