Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: runner-gearpump
    • Labels:
      None

      Description

      Intel is submitting Gearpump (http://www.gearpump.io) to ASF (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of low-level primitives a la MillWheel, with some higher level primitives like non-merging windowing mixed in. Seems like it would make a nice Beam runner.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          mauzhang Manu Zhang added a comment -

          This is manuzhang from Gearpump team. May I take this task ?

          Show
          mauzhang Manu Zhang added a comment - This is manuzhang from Gearpump team. May I take this task ?
          Hide
          frances Frances Perry added a comment -

          Happy to assign it to you, as you will clearly be the expert on Gearpump

          But please note that Beam is still very much under construction and there are a number of breaking API changes likely in the near future. So please reach out before getting beyond the early design phase / determining how well the models align. If you haven't yet, I'd start with these resources: http://beam.incubator.apache.org/getting_started/

          Show
          frances Frances Perry added a comment - Happy to assign it to you, as you will clearly be the expert on Gearpump But please note that Beam is still very much under construction and there are a number of breaking API changes likely in the near future. So please reach out before getting beyond the early design phase / determining how well the models align. If you haven't yet, I'd start with these resources: http://beam.incubator.apache.org/getting_started/
          Hide
          mauzhang Manu Zhang added a comment -

          Thanks. I'm glad to take this task, and the Gearpump team is also very excited. I'll follow the guides and reach out for you guys whenever in doubts.

          Show
          mauzhang Manu Zhang added a comment - Thanks. I'm glad to take this task, and the Gearpump team is also very excited. I'll follow the guides and reach out for you guys whenever in doubts.
          Hide
          darion darion yaphet added a comment -

          Apache Beam is both batch and streaming process and Gearpump is focus on streaming processing .

          Show
          darion darion yaphet added a comment - Apache Beam is both batch and streaming process and Gearpump is focus on streaming processing .
          Hide
          takidau Tyler Akidau added a comment -

          That's fine. Batch is essentially a subset of streaming from a semantics perspective. And even if it weren't, we don't require all runners to implement all of the features in the model; runners are allowed to implement only the pieces that make sense for them.

          Show
          takidau Tyler Akidau added a comment - That's fine. Batch is essentially a subset of streaming from a semantics perspective. And even if it weren't, we don't require all runners to implement all of the features in the model; runners are allowed to implement only the pieces that make sense for them.
          Hide
          mauzhang Manu Zhang added a comment -

          Here is a design document . Please kindly comment it.

          Show
          mauzhang Manu Zhang added a comment - Here is a design document . Please kindly comment it.
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          It sounds good to me. Thanks Manu !

          Show
          jbonofre Jean-Baptiste Onofré added a comment - It sounds good to me. Thanks Manu !
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/incubator-beam/pull/323

          BEAM-79 add Gearpump runner

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          This PR adds Gearpump runner to Beam meeting the goals of phase 1 in the [design document](https://docs.google.com/document/d/1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g/edit).

          The Gearpump runner supports the following functionalities,

          • Transformations: ParDo, GroupByKey, Flatten
          • Windows: using Beam's window logic and code
          • side outputs
          • serialization/deserialization: using Gearpump's Kryo serializer
          • sources: Beam's UnboundedSource
          • message delivery guarantee: at-most-once
          • tests: integration test for various translators

          Here's a snapshot of running the following Beam example on Gearpump cluster

          ```java
          PCollection<KV<String, Long>> wordCounts =
          p.apply(Read.from(new UnboundedTextSource()).named("WordStream"))
          .apply(ParDo.of(new ExtractWordsFn()))
          .apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(10))))
          .apply(Count.<String>perElement());

          wordCounts.apply(ParDo.of(new FormatAsStringFn()));
          ```

          ![snip20160511_4](https://cloud.githubusercontent.com/assets/1191767/15171197/fd6ffba0-177e-11e6-99a1-30c7c2597244.png)

          Note that the Gearpump runner is still in early stage and lacking capabilities like trigger, side inputs, aggregator. However, I'd like to have the community to get a feel of what Gearpump is like, whether Beam and Gearpump go well, and gather ideas for improvements.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump_runner

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/323.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #323


          commit 73e5978f599bcf32ed8c2f1d54b6bd3bd8350092
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-03-15T08:15:16Z

          BEAM-79 add Gearpump runner


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-beam/pull/323 BEAM-79 add Gearpump runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — This PR adds Gearpump runner to Beam meeting the goals of phase 1 in the [design document] ( https://docs.google.com/document/d/1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g/edit ). The Gearpump runner supports the following functionalities, Transformations: ParDo, GroupByKey, Flatten Windows: using Beam's window logic and code side outputs serialization/deserialization: using Gearpump's Kryo serializer sources: Beam's UnboundedSource message delivery guarantee: at-most-once tests: integration test for various translators Here's a snapshot of running the following Beam example on Gearpump cluster ```java PCollection<KV<String, Long>> wordCounts = p.apply(Read.from(new UnboundedTextSource()).named("WordStream")) .apply(ParDo.of(new ExtractWordsFn())) .apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(10)))) .apply(Count.<String>perElement()); wordCounts.apply(ParDo.of(new FormatAsStringFn())); ``` ! [snip20160511_4] ( https://cloud.githubusercontent.com/assets/1191767/15171197/fd6ffba0-177e-11e6-99a1-30c7c2597244.png ) Note that the Gearpump runner is still in early stage and lacking capabilities like trigger, side inputs, aggregator. However, I'd like to have the community to get a feel of what Gearpump is like, whether Beam and Gearpump go well, and gather ideas for improvements. You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump_runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/323.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #323 commit 73e5978f599bcf32ed8c2f1d54b6bd3bd8350092 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-03-15T08:15:16Z BEAM-79 add Gearpump runner
          Hide
          jbonofre Jean-Baptiste Onofré added a comment -

          Starting to review and test. Thanks !

          Show
          jbonofre Jean-Baptiste Onofré added a comment - Starting to review and test. Thanks !
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/incubator-beam/pull/323

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/incubator-beam/pull/323
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kennknowles opened a pull request:

          https://github.com/apache/incubator-beam/pull/714

          BEAM-79 Add Gearpump runner to runners parent pom.xml

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [x] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [x] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [x] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          I think this line got lost in the shuffle. It is why we aren't seeing the module in the Jenkins web UI, though the [tests are running and passing](https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/).

          R: @dhalperi
          CC: @manuzhang

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/kennknowles/incubator-beam gearpump-module

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/714.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #714


          commit f5e1abb61ea36866cd2d15be6e82c2b0785a4d9c
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-22T19:50:43Z

          Add Gearpump runner to runners parent pom.xml


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/714 BEAM-79 Add Gearpump runner to runners parent pom.xml Be sure to do all of the following to help us incorporate your contribution quickly and easily: [x] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [x] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — I think this line got lost in the shuffle. It is why we aren't seeing the module in the Jenkins web UI, though the [tests are running and passing] ( https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/ ). R: @dhalperi CC: @manuzhang You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam gearpump-module Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/714.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #714 commit f5e1abb61ea36866cd2d15be6e82c2b0785a4d9c Author: Kenneth Knowles <klk@google.com> Date: 2016-07-22T19:50:43Z Add Gearpump runner to runners parent pom.xml
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kennknowles closed the pull request at:

          https://github.com/apache/incubator-beam/pull/714

          Show
          githubbot ASF GitHub Bot added a comment - Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/714
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/incubator-beam/pull/750

          BEAM-79 Merge branch 'master' into gearpump_runner

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump_runner_sync

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/750.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #750


          commit 34467f92d5a31b47f95b734c737fde0a8277311b
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-15T17:51:24Z

          Make TransformEvaluatorFactory reuse Explicit

          Transform Evaluator Factories must be reused for the entire execution of
          a Pipeline and must not be reused across pipelines.

          Remove EvaluatorKey, and key explicitly by the transform application.

          commit ad6ee01825740ee47f25ca036fa5f352375bbe6e
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-19T06:40:13Z

          Checkstyle: enforce package-info.java presence

          Ignores tests and examples

          commit c4ad11832235eef0b73d299cd02d1a224c130ece
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-20T16:16:28Z

          Closes #692

          commit 6d7efe3df8cde590722d94441eca3922a3a67734
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-20T17:30:55Z

          This closes #666

          commit 5f19e4caf207ff02d226fec2933be2f37ca66b4d
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-06-30T17:06:52Z

          Add withAllowedLateness with Closing Behavior to Window

          This makes the static constructors for withAllowedLateness symmetric to
          the PTransform builder methods. It also allows references to
          Window#withAllowedLateness(Duration, ClosingBehavior).

          commit f547f70e1c9535ff663d124f67c72c7ec2c55e9e
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-20T17:40:13Z

          This closes #567

          commit 00195d2543eb347cc3669a4ac89e98da0bc4dca4
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-06-28T22:44:49Z

          Use the ParDo Application to Cache DoFns

          A DoFn application is the scope of reuse.

          Factor CloningThreadLocal as the top-level class instead of
          SerializableCloningThreadLocalCacheLoader, and extract the Fn from the
          AppliedPTransform when loading an absent element.

          commit 436e4a34ebb222545cb03cb6d39ea4ca2d905254
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-20T17:55:53Z

          This closes #554

          commit b240525affb205a83054577233f3a4a508fe1c72
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-18T19:05:02Z

          BigQueryIO: move to google-cloud-platform module

          • Move package from io to io.gcp.bigquery
          • Move from SDK core into GCP-IO module
          • Fixup references and import orders
          • Separate AvroUtils into generic AvroUtils and BigQueryAvroUtils
          • Rewrite a unit test in sdk core to not depend on BigQueryIO
          • Fixup Javadoc in SDK core that need not depend on BigQueryIO
          • Make utility classes package-private

          commit 7ec8781a2e18548a23c882329f0b50f7254202ec
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-20T20:02:43Z

          Closes #681

          commit bdb65278873a5010a625dc6a569ba25b17374c06
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-20T04:36:10Z

          Add os-maven-plugin to Spark runner

          commit 84332ee9716233af928e85c14c534714ab828531
          Author: Chandni Singh <chandni.singh@capitalone.com>
          Date: 2016-07-20T00:30:16Z

          BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes

          commit 6d5e8186a2da532eb1c29097bc1259a19d9f72c9
          Author: Luke Cwik <lcwik@visitor-lcwik.wat.corp.google.com>
          Date: 2016-07-21T13:34:52Z

          BEAM-372 added a test that verifies if a coder consumes bytes equal to encoded bytes

          This closes #695

          commit ae2144196c351cc2ee544e030d793929d8607696
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-20T17:26:02Z

          BigtableIO: upgrade to 0.9.1

          • Use the uber jar
          • Remove OS classifier mumbo jumbo
          • Move common dependency versioning to root pom

          commit 0eb81fdc15bc3dce28e24e73427bd389e0f6d60d
          Author: Luke Cwik <lcwik@visitor-lcwik.wat.corp.google.com>
          Date: 2016-07-21T15:26:41Z

          BEAM-445 BigtableIO: upgrade to 0.9.1

          This closes #701

          commit 232c0e9610a5d5559598a2059577b699299a0b76
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-20T16:49:47Z

          Rename DirectRegistrar Registrars

          This removes the duplication of "DirectRunner" and "DirectOptions"
          classes.

          commit 9cdaed0d8a401caf5a9efb626fa629fd6f4f4627
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-21T17:24:54Z

          Closes #700

          commit 7337ecf147ca69262978db116a5a1f739aca28ae
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2016-05-31T16:04:52Z

          BEAM-316 Add file scheme support in TextIO

          commit 38061d3531a1ca13ef825a6ee50872e420ab1494
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-18T22:35:06Z

          BEAM-316 Add file URI handling

          • Register FileIOChannelFactory for file scheme
          • Modify FileIOChannelFactory to dynamically remove the file:// scheme string.

          commit adb472083460bc175a0685133eadb6a336f1a4a2
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-22T03:17:54Z

          Closes #684 and #402

          commit 060bc802f0bff8ca1a076e9f93a623058e1d651b
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-22T04:01:36Z

          Execute RunnableOnService tests only when runner options provided

          Previously, the situation was this:

          • All runners inherit a RunnableOnService integration-test
            execution referencing runnableOnServicePipelineOptions
            whether or not the variable was set. Basically an unbound
            variable reference.
          • The Dataflow runner had a profile disabling it if
            runnableOnServicePipelineOptions was not set.
          • Before they got configured, Flink and Spark had to
            do extra work to explicitly prevent the invalid
            configuration from being used.

          After this change:

          • All runners inherit the same integration-test execution
            but only if the variable it requires is present.
          • Dataflow doesn't have any special profile.
          • Flink and Spark are unchanged, since they do set
            up the variable themselves. When they move to running
            only as postcommit, like Dataflow does, the hardcoding
            is expected to either move to a profile or move to
            the Jenkins invocation.

          commit 07a313f1af78349040afd9ff6b01e60cdcd567a3
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-22T07:39:59Z

          Closes #709

          commit 75df931a55dbd3f93d9143574cdad7291d3457aa
          Author: Jesse Anderson <jesse@smokinghand.com>
          Date: 2016-07-22T20:54:26Z

          Added BigInteger to TypeDescriptors class.

          commit 12b60ffa5d18812674299a97e84af31d063e170c
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-22T23:51:37Z

          Closes #716

          commit bf3af5de9e6f8126ffd2ccd3f7a68a84e55e90ff
          Author: Pei He <peihe@google.com>
          Date: 2016-07-22T22:03:38Z

          BEAM-480 Move insertAll() from BigQueryTableInserter to BigQueryServices

          commit 122cd04663529f4fc44530ed9e144e0dfd68360f
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-22T23:52:50Z

          Closes #717

          commit 12fb77811d7bf8c72a3c0a48f391bab3c815fbef
          Author: Sela <ansela@paypal.com>
          Date: 2016-06-27T17:59:07Z

          Remove dependency on beam-examples-java.

          Duplicate WordCount into spark examlpes package.

          Duplicate parts of TfIdf from beam examlpes.

          Better reuse of WordCount and its parts.

          Remove dependency on beam-examples-java

          commit d3dc368666d5b700c5026fe08455e1210692ce3e
          Author: Sela <ansela@paypal.com>
          Date: 2016-07-04T12:33:57Z

          Remove JavaDoc links to beam-examples.

          commit cf1464465e1e65be2e4896cc6f693f8bbe82b844
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-07-23T02:51:50Z

          Closes #539

          commit 2fe387707d1e115b578f5ee643bb99c0e4667ee0
          Author: Maximilian Michels <mxm@apache.org>
          Date: 2016-07-20T14:06:06Z

          [flink] improve example section in README

          • updates the README
          • repairs broken exec configuration

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-beam/pull/750 BEAM-79 Merge branch 'master' into gearpump_runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump_runner_sync Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/750.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #750 commit 34467f92d5a31b47f95b734c737fde0a8277311b Author: Thomas Groh <tgroh@google.com> Date: 2016-07-15T17:51:24Z Make TransformEvaluatorFactory reuse Explicit Transform Evaluator Factories must be reused for the entire execution of a Pipeline and must not be reused across pipelines. Remove EvaluatorKey, and key explicitly by the transform application. commit ad6ee01825740ee47f25ca036fa5f352375bbe6e Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-19T06:40:13Z Checkstyle: enforce package-info.java presence Ignores tests and examples commit c4ad11832235eef0b73d299cd02d1a224c130ece Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-20T16:16:28Z Closes #692 commit 6d7efe3df8cde590722d94441eca3922a3a67734 Author: Kenneth Knowles <klk@google.com> Date: 2016-07-20T17:30:55Z This closes #666 commit 5f19e4caf207ff02d226fec2933be2f37ca66b4d Author: Thomas Groh <tgroh@google.com> Date: 2016-06-30T17:06:52Z Add withAllowedLateness with Closing Behavior to Window This makes the static constructors for withAllowedLateness symmetric to the PTransform builder methods. It also allows references to Window#withAllowedLateness(Duration, ClosingBehavior). commit f547f70e1c9535ff663d124f67c72c7ec2c55e9e Author: Kenneth Knowles <klk@google.com> Date: 2016-07-20T17:40:13Z This closes #567 commit 00195d2543eb347cc3669a4ac89e98da0bc4dca4 Author: Thomas Groh <tgroh@google.com> Date: 2016-06-28T22:44:49Z Use the ParDo Application to Cache DoFns A DoFn application is the scope of reuse. Factor CloningThreadLocal as the top-level class instead of SerializableCloningThreadLocalCacheLoader, and extract the Fn from the AppliedPTransform when loading an absent element. commit 436e4a34ebb222545cb03cb6d39ea4ca2d905254 Author: Kenneth Knowles <klk@google.com> Date: 2016-07-20T17:55:53Z This closes #554 commit b240525affb205a83054577233f3a4a508fe1c72 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-18T19:05:02Z BigQueryIO: move to google-cloud-platform module Move package from io to io.gcp.bigquery Move from SDK core into GCP-IO module Fixup references and import orders Separate AvroUtils into generic AvroUtils and BigQueryAvroUtils Rewrite a unit test in sdk core to not depend on BigQueryIO Fixup Javadoc in SDK core that need not depend on BigQueryIO Make utility classes package-private commit 7ec8781a2e18548a23c882329f0b50f7254202ec Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-20T20:02:43Z Closes #681 commit bdb65278873a5010a625dc6a569ba25b17374c06 Author: Kenneth Knowles <klk@google.com> Date: 2016-07-20T04:36:10Z Add os-maven-plugin to Spark runner commit 84332ee9716233af928e85c14c534714ab828531 Author: Chandni Singh <chandni.singh@capitalone.com> Date: 2016-07-20T00:30:16Z BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes commit 6d5e8186a2da532eb1c29097bc1259a19d9f72c9 Author: Luke Cwik <lcwik@visitor-lcwik.wat.corp.google.com> Date: 2016-07-21T13:34:52Z BEAM-372 added a test that verifies if a coder consumes bytes equal to encoded bytes This closes #695 commit ae2144196c351cc2ee544e030d793929d8607696 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-20T17:26:02Z BigtableIO: upgrade to 0.9.1 Use the uber jar Remove OS classifier mumbo jumbo Move common dependency versioning to root pom commit 0eb81fdc15bc3dce28e24e73427bd389e0f6d60d Author: Luke Cwik <lcwik@visitor-lcwik.wat.corp.google.com> Date: 2016-07-21T15:26:41Z BEAM-445 BigtableIO: upgrade to 0.9.1 This closes #701 commit 232c0e9610a5d5559598a2059577b699299a0b76 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-20T16:49:47Z Rename DirectRegistrar Registrars This removes the duplication of "DirectRunner" and "DirectOptions" classes. commit 9cdaed0d8a401caf5a9efb626fa629fd6f4f4627 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-21T17:24:54Z Closes #700 commit 7337ecf147ca69262978db116a5a1f739aca28ae Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2016-05-31T16:04:52Z BEAM-316 Add file scheme support in TextIO commit 38061d3531a1ca13ef825a6ee50872e420ab1494 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-18T22:35:06Z BEAM-316 Add file URI handling Register FileIOChannelFactory for file scheme Modify FileIOChannelFactory to dynamically remove the file:// scheme string. commit adb472083460bc175a0685133eadb6a336f1a4a2 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-22T03:17:54Z Closes #684 and #402 commit 060bc802f0bff8ca1a076e9f93a623058e1d651b Author: Kenneth Knowles <klk@google.com> Date: 2016-07-22T04:01:36Z Execute RunnableOnService tests only when runner options provided Previously, the situation was this: All runners inherit a RunnableOnService integration-test execution referencing runnableOnServicePipelineOptions whether or not the variable was set. Basically an unbound variable reference. The Dataflow runner had a profile disabling it if runnableOnServicePipelineOptions was not set. Before they got configured, Flink and Spark had to do extra work to explicitly prevent the invalid configuration from being used. After this change: All runners inherit the same integration-test execution but only if the variable it requires is present. Dataflow doesn't have any special profile. Flink and Spark are unchanged, since they do set up the variable themselves. When they move to running only as postcommit, like Dataflow does, the hardcoding is expected to either move to a profile or move to the Jenkins invocation. commit 07a313f1af78349040afd9ff6b01e60cdcd567a3 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-22T07:39:59Z Closes #709 commit 75df931a55dbd3f93d9143574cdad7291d3457aa Author: Jesse Anderson <jesse@smokinghand.com> Date: 2016-07-22T20:54:26Z Added BigInteger to TypeDescriptors class. commit 12b60ffa5d18812674299a97e84af31d063e170c Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-22T23:51:37Z Closes #716 commit bf3af5de9e6f8126ffd2ccd3f7a68a84e55e90ff Author: Pei He <peihe@google.com> Date: 2016-07-22T22:03:38Z BEAM-480 Move insertAll() from BigQueryTableInserter to BigQueryServices commit 122cd04663529f4fc44530ed9e144e0dfd68360f Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-22T23:52:50Z Closes #717 commit 12fb77811d7bf8c72a3c0a48f391bab3c815fbef Author: Sela <ansela@paypal.com> Date: 2016-06-27T17:59:07Z Remove dependency on beam-examples-java. Duplicate WordCount into spark examlpes package. Duplicate parts of TfIdf from beam examlpes. Better reuse of WordCount and its parts. Remove dependency on beam-examples-java commit d3dc368666d5b700c5026fe08455e1210692ce3e Author: Sela <ansela@paypal.com> Date: 2016-07-04T12:33:57Z Remove JavaDoc links to beam-examples. commit cf1464465e1e65be2e4896cc6f693f8bbe82b844 Author: Dan Halperin <dhalperi@google.com> Date: 2016-07-23T02:51:50Z Closes #539 commit 2fe387707d1e115b578f5ee643bb99c0e4667ee0 Author: Maximilian Michels <mxm@apache.org> Date: 2016-07-20T14:06:06Z [flink] improve example section in README updates the README repairs broken exec configuration
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/incubator-beam/pull/750

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/incubator-beam/pull/750
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/incubator-beam/pull/943

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/incubator-beam/pull/943
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/incubator-beam/pull/956

          BEAM-79 fix integration-test failure

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/956.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #956


          commit 272fe9f6eb6c721a1d4a39cc816e3de9dc705373
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-09-14T03:20:27Z

          BEAM-79 fix integration-test failure


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-beam/pull/956 BEAM-79 fix integration-test failure Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/956.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #956 commit 272fe9f6eb6c721a1d4a39cc816e3de9dc705373 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-09-14T03:20:27Z BEAM-79 fix integration-test failure
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/incubator-beam/pull/956

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/956
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/incubator-beam/pull/959

          BEAM-79 upgrade surefire version to exclude tests

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/959.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #959


          commit c879a7fa02259458029d8dca82ecaf80d36019b7
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-09-14T22:52:09Z

          BEAM-79 upgrade surefire version to exclude tests


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-beam/pull/959 BEAM-79 upgrade surefire version to exclude tests Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/959.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #959 commit c879a7fa02259458029d8dca82ecaf80d36019b7 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-09-14T22:52:09Z BEAM-79 upgrade surefire version to exclude tests
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/incubator-beam/pull/959

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/incubator-beam/pull/959
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/incubator-beam/pull/1193

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1193
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/incubator-beam/pull/1234

          BEAM-79 port OldDoFn to new DoFn

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-v2

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/1234.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1234


          commit ccee98f39236062dd501b63d9a89e88dd1fd47bd
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-10-31T03:52:22Z

          BEAM-79 port OldDoFn to new DoFn


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-beam/pull/1234 BEAM-79 port OldDoFn to new DoFn Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1234.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1234 commit ccee98f39236062dd501b63d9a89e88dd1fd47bd Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-10-31T03:52:22Z BEAM-79 port OldDoFn to new DoFn
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/incubator-beam/pull/1234

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/incubator-beam/pull/1234
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/incubator-beam/pull/1306

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1306
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/incubator-beam/pull/1663

          BEAM-79 merge master into gearpump-runner branch

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/1663.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1663


          commit 0af145bba721f7cdf3fbcd15330df433d5a8431d
          Author: Davor Bonaci <davor@google.com>
          Date: 2016-12-07T01:00:31Z

          This closes #1504

          commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
          Author: Jakob Homan <jghoman@gmail.com>
          Date: 2016-12-07T00:59:50Z

          BEAM-1099 Minor typos in KafkaIO

          commit afedd68e806830549724dfc0f2565d756cc6b46d
          Author: Davor Bonaci <davor@google.com>
          Date: 2016-12-07T01:03:54Z

          This closes #1524

          commit e8c9686a2e898d38afd692328eb171c542084747
          Author: Pei He <peihe@google.com>
          Date: 2016-11-23T23:59:56Z

          BEAM-1047 Add DataflowClient wrapper on top of JSON library.

          commit ded58832ceaef487f4590d9396f09744288c955d
          Author: Pei He <peihe@google.com>
          Date: 2016-11-24T00:14:27Z

          [Code Health] Remove redundant projectId from DataflowPipelineJob.

          commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
          Author: Pei He <peihe@google.com>
          Date: 2016-11-28T19:47:42Z

          BEAM-1047 Update dataflow runner code to use DataflowClient wrapper.

          commit b2b570f27808b1671bf6cd0fc60f874da671d4ca
          Author: bchambers <bchambers@google.com>
          Date: 2016-12-07T01:08:13Z

          Closes #1434

          commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152
          Author: Sam McVeety <sgmc@google.com>
          Date: 2016-10-30T18:58:44Z

          BigQueryIO.Read: support runtime options

          commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-07T05:28:13Z

          FLINK-1102 Fix Aggregator Registration in Flink Batch Runner

          commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-07T07:26:02Z

          This closes #1530

          commit baf5e6bd9b1011f4c5c3974aa46393471b340c15
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2016-12-07T07:37:33Z

          BEAM-1094 Set test scope for Kafka IO and junit

          commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f
          Author: Sela <ansela@paypal.com>
          Date: 2016-12-07T08:31:38Z

          This closes #1531

          commit dce3a196a3a26fdd42225520faf3d9084ee48123
          Author: Sela <ansela@paypal.com>
          Date: 2016-12-07T09:20:07Z

          BEAM-329 Update Spark runner README.

          commit 02bb8c375c48847b1686d70184fb194500a62e8c
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2016-12-07T11:51:09Z

          BEAM-329 This closes #1532

          commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-06T23:20:28Z

          Port most of DoFnRunner Javadoc to new DoFn

          commit 1526184ae8be1f8ae6863f830653204157a584cd
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-12-07T16:51:02Z

          This closes #1527

          commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T05:01:37Z

          Port ParDoTest from OldDoFn to new DoFn

          commit ae52ec1bc3f3120e9f8e150e50c18564055a467c
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T17:00:18Z

          This closes #1529

          commit 55d333bff68809ff1a9154491ace02d2d16e3b85
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-12-05T22:29:05Z

          Only provide expanded Inputs and Outputs

          This removes PInput and POutput from the immediate API Surface of
          TransformHierarchy.Node, and forces Pipeline Visitors to access only
          the expanded version of the output.

          This is part of the move towards the runner-agnostic representation of a
          graph.

          commit 5b31a369962907e257de8019fbf6cde4c615b1c0
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-12-07T17:14:38Z

          This closes #1511

          commit 43fef2775145f67def3ab8a246ecca192a7d650b
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-07T12:06:57Z

          BEAM-905 Add shading config to examples archetype and enable it for Flink

          This makes the Flink quickstart work out of the box.

          commit b44a7ac4a2a0ac9fd7a8e38cc5eea5073c629085
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-07T22:55:03Z

          Closes #1533

          commit 6439f701d1008d6a0432828e11e0fcc8a4fe6ecc
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-07T23:40:58Z

          BEAM-1108 DataflowRunner: remove deprecated TEARDOWN_POLICY control

          commit 3b2e0290ddbedd199926a70f36c02ea4515841cb
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-08T01:18:11Z

          Closes #1546

          commit 1b1252074dd6b57f4fb88ceb82c704d3d3d8147f
          Author: Alexey Diomin <diominay@gmail.com>
          Date: 2016-12-07T05:39:27Z

          BEAM-1095 Add support set config for reuse-object on flink

          commit c53e0b1623e3ee3c08c329e2716440f031681591
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-08T01:44:07Z

          This closes #1518

          commit f1a5704a505b01d7d4649b61d1f6697859367964
          Author: Alexey Diomin <diominay@gmail.com>
          Date: 2016-12-07T05:48:35Z

          BEAM-1096 Flink streaming side output optimization using SplitStream

          commit 6807480a97f2315b3f48ad8dd5accb4e30475fa4
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-08T01:55:32Z

          This closes #1520

          commit 3f8c80769a3bb38da64c6628fd8c4669fcac794b
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T04:10:06Z

          Add DoFn.OnTimerContext

          commit 2883062eebe8dba849ab89627f6aeb53266ac1a8
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T04:10:21Z

          Access to OnTimerContext via DoFnInvokers.ArgumentProvider


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-beam/pull/1663 BEAM-79 merge master into gearpump-runner branch Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1663.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1663 commit 0af145bba721f7cdf3fbcd15330df433d5a8431d Author: Davor Bonaci <davor@google.com> Date: 2016-12-07T01:00:31Z This closes #1504 commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f Author: Jakob Homan <jghoman@gmail.com> Date: 2016-12-07T00:59:50Z BEAM-1099 Minor typos in KafkaIO commit afedd68e806830549724dfc0f2565d756cc6b46d Author: Davor Bonaci <davor@google.com> Date: 2016-12-07T01:03:54Z This closes #1524 commit e8c9686a2e898d38afd692328eb171c542084747 Author: Pei He <peihe@google.com> Date: 2016-11-23T23:59:56Z BEAM-1047 Add DataflowClient wrapper on top of JSON library. commit ded58832ceaef487f4590d9396f09744288c955d Author: Pei He <peihe@google.com> Date: 2016-11-24T00:14:27Z [Code Health] Remove redundant projectId from DataflowPipelineJob. commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a Author: Pei He <peihe@google.com> Date: 2016-11-28T19:47:42Z BEAM-1047 Update dataflow runner code to use DataflowClient wrapper. commit b2b570f27808b1671bf6cd0fc60f874da671d4ca Author: bchambers <bchambers@google.com> Date: 2016-12-07T01:08:13Z Closes #1434 commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152 Author: Sam McVeety <sgmc@google.com> Date: 2016-10-30T18:58:44Z BigQueryIO.Read: support runtime options commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-07T05:28:13Z FLINK-1102 Fix Aggregator Registration in Flink Batch Runner commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-07T07:26:02Z This closes #1530 commit baf5e6bd9b1011f4c5c3974aa46393471b340c15 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2016-12-07T07:37:33Z BEAM-1094 Set test scope for Kafka IO and junit commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f Author: Sela <ansela@paypal.com> Date: 2016-12-07T08:31:38Z This closes #1531 commit dce3a196a3a26fdd42225520faf3d9084ee48123 Author: Sela <ansela@paypal.com> Date: 2016-12-07T09:20:07Z BEAM-329 Update Spark runner README. commit 02bb8c375c48847b1686d70184fb194500a62e8c Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2016-12-07T11:51:09Z BEAM-329 This closes #1532 commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f Author: Kenneth Knowles <klk@google.com> Date: 2016-12-06T23:20:28Z Port most of DoFnRunner Javadoc to new DoFn commit 1526184ae8be1f8ae6863f830653204157a584cd Author: Thomas Groh <tgroh@google.com> Date: 2016-12-07T16:51:02Z This closes #1527 commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4 Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T05:01:37Z Port ParDoTest from OldDoFn to new DoFn commit ae52ec1bc3f3120e9f8e150e50c18564055a467c Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T17:00:18Z This closes #1529 commit 55d333bff68809ff1a9154491ace02d2d16e3b85 Author: Thomas Groh <tgroh@google.com> Date: 2016-12-05T22:29:05Z Only provide expanded Inputs and Outputs This removes PInput and POutput from the immediate API Surface of TransformHierarchy.Node, and forces Pipeline Visitors to access only the expanded version of the output. This is part of the move towards the runner-agnostic representation of a graph. commit 5b31a369962907e257de8019fbf6cde4c615b1c0 Author: Thomas Groh <tgroh@google.com> Date: 2016-12-07T17:14:38Z This closes #1511 commit 43fef2775145f67def3ab8a246ecca192a7d650b Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-07T12:06:57Z BEAM-905 Add shading config to examples archetype and enable it for Flink This makes the Flink quickstart work out of the box. commit b44a7ac4a2a0ac9fd7a8e38cc5eea5073c629085 Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-07T22:55:03Z Closes #1533 commit 6439f701d1008d6a0432828e11e0fcc8a4fe6ecc Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-07T23:40:58Z BEAM-1108 DataflowRunner: remove deprecated TEARDOWN_POLICY control commit 3b2e0290ddbedd199926a70f36c02ea4515841cb Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-08T01:18:11Z Closes #1546 commit 1b1252074dd6b57f4fb88ceb82c704d3d3d8147f Author: Alexey Diomin <diominay@gmail.com> Date: 2016-12-07T05:39:27Z BEAM-1095 Add support set config for reuse-object on flink commit c53e0b1623e3ee3c08c329e2716440f031681591 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-08T01:44:07Z This closes #1518 commit f1a5704a505b01d7d4649b61d1f6697859367964 Author: Alexey Diomin <diominay@gmail.com> Date: 2016-12-07T05:48:35Z BEAM-1096 Flink streaming side output optimization using SplitStream commit 6807480a97f2315b3f48ad8dd5accb4e30475fa4 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-08T01:55:32Z This closes #1520 commit 3f8c80769a3bb38da64c6628fd8c4669fcac794b Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T04:10:06Z Add DoFn.OnTimerContext commit 2883062eebe8dba849ab89627f6aeb53266ac1a8 Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T04:10:21Z Access to OnTimerContext via DoFnInvokers.ArgumentProvider
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/incubator-beam/pull/1663

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/incubator-beam/pull/1663
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang reopened a pull request:

          https://github.com/apache/incubator-beam/pull/1663

          BEAM-79 merge master into gearpump-runner branch

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [x] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [x] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [x] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-beam/pull/1663.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1663


          commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
          Author: Jakob Homan <jghoman@gmail.com>
          Date: 2016-12-07T00:59:50Z

          BEAM-1099 Minor typos in KafkaIO

          commit afedd68e806830549724dfc0f2565d756cc6b46d
          Author: Davor Bonaci <davor@google.com>
          Date: 2016-12-07T01:03:54Z

          This closes #1524

          commit e8c9686a2e898d38afd692328eb171c542084747
          Author: Pei He <peihe@google.com>
          Date: 2016-11-23T23:59:56Z

          BEAM-1047 Add DataflowClient wrapper on top of JSON library.

          commit ded58832ceaef487f4590d9396f09744288c955d
          Author: Pei He <peihe@google.com>
          Date: 2016-11-24T00:14:27Z

          [Code Health] Remove redundant projectId from DataflowPipelineJob.

          commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
          Author: Pei He <peihe@google.com>
          Date: 2016-11-28T19:47:42Z

          BEAM-1047 Update dataflow runner code to use DataflowClient wrapper.

          commit b2b570f27808b1671bf6cd0fc60f874da671d4ca
          Author: bchambers <bchambers@google.com>
          Date: 2016-12-07T01:08:13Z

          Closes #1434

          commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152
          Author: Sam McVeety <sgmc@google.com>
          Date: 2016-10-30T18:58:44Z

          BigQueryIO.Read: support runtime options

          commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-07T05:28:13Z

          FLINK-1102 Fix Aggregator Registration in Flink Batch Runner

          commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-07T07:26:02Z

          This closes #1530

          commit baf5e6bd9b1011f4c5c3974aa46393471b340c15
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2016-12-07T07:37:33Z

          BEAM-1094 Set test scope for Kafka IO and junit

          commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f
          Author: Sela <ansela@paypal.com>
          Date: 2016-12-07T08:31:38Z

          This closes #1531

          commit dce3a196a3a26fdd42225520faf3d9084ee48123
          Author: Sela <ansela@paypal.com>
          Date: 2016-12-07T09:20:07Z

          BEAM-329 Update Spark runner README.

          commit 02bb8c375c48847b1686d70184fb194500a62e8c
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2016-12-07T11:51:09Z

          BEAM-329 This closes #1532

          commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-06T23:20:28Z

          Port most of DoFnRunner Javadoc to new DoFn

          commit 1526184ae8be1f8ae6863f830653204157a584cd
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-12-07T16:51:02Z

          This closes #1527

          commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T05:01:37Z

          Port ParDoTest from OldDoFn to new DoFn

          commit ae52ec1bc3f3120e9f8e150e50c18564055a467c
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T17:00:18Z

          This closes #1529

          commit 55d333bff68809ff1a9154491ace02d2d16e3b85
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-12-05T22:29:05Z

          Only provide expanded Inputs and Outputs

          This removes PInput and POutput from the immediate API Surface of
          TransformHierarchy.Node, and forces Pipeline Visitors to access only
          the expanded version of the output.

          This is part of the move towards the runner-agnostic representation of a
          graph.

          commit 5b31a369962907e257de8019fbf6cde4c615b1c0
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-12-07T17:14:38Z

          This closes #1511

          commit 43fef2775145f67def3ab8a246ecca192a7d650b
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-07T12:06:57Z

          BEAM-905 Add shading config to examples archetype and enable it for Flink

          This makes the Flink quickstart work out of the box.

          commit b44a7ac4a2a0ac9fd7a8e38cc5eea5073c629085
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-07T22:55:03Z

          Closes #1533

          commit 6439f701d1008d6a0432828e11e0fcc8a4fe6ecc
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-07T23:40:58Z

          BEAM-1108 DataflowRunner: remove deprecated TEARDOWN_POLICY control

          commit 3b2e0290ddbedd199926a70f36c02ea4515841cb
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-12-08T01:18:11Z

          Closes #1546

          commit 1b1252074dd6b57f4fb88ceb82c704d3d3d8147f
          Author: Alexey Diomin <diominay@gmail.com>
          Date: 2016-12-07T05:39:27Z

          BEAM-1095 Add support set config for reuse-object on flink

          commit c53e0b1623e3ee3c08c329e2716440f031681591
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-08T01:44:07Z

          This closes #1518

          commit f1a5704a505b01d7d4649b61d1f6697859367964
          Author: Alexey Diomin <diominay@gmail.com>
          Date: 2016-12-07T05:48:35Z

          BEAM-1096 Flink streaming side output optimization using SplitStream

          commit 6807480a97f2315b3f48ad8dd5accb4e30475fa4
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2016-12-08T01:55:32Z

          This closes #1520

          commit 3f8c80769a3bb38da64c6628fd8c4669fcac794b
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T04:10:06Z

          Add DoFn.OnTimerContext

          commit 2883062eebe8dba849ab89627f6aeb53266ac1a8
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T04:10:21Z

          Access to OnTimerContext via DoFnInvokers.ArgumentProvider

          commit 42b506f06dbd73e03a2cfad4e7677e9698b3c020
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-12-07T04:18:18Z

          Add OnTimerContext parameter support to DoFnSignature


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang reopened a pull request: https://github.com/apache/incubator-beam/pull/1663 BEAM-79 merge master into gearpump-runner branch Be sure to do all of the following to help us incorporate your contribution quickly and easily: [x] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [x] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1663.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1663 commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f Author: Jakob Homan <jghoman@gmail.com> Date: 2016-12-07T00:59:50Z BEAM-1099 Minor typos in KafkaIO commit afedd68e806830549724dfc0f2565d756cc6b46d Author: Davor Bonaci <davor@google.com> Date: 2016-12-07T01:03:54Z This closes #1524 commit e8c9686a2e898d38afd692328eb171c542084747 Author: Pei He <peihe@google.com> Date: 2016-11-23T23:59:56Z BEAM-1047 Add DataflowClient wrapper on top of JSON library. commit ded58832ceaef487f4590d9396f09744288c955d Author: Pei He <peihe@google.com> Date: 2016-11-24T00:14:27Z [Code Health] Remove redundant projectId from DataflowPipelineJob. commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a Author: Pei He <peihe@google.com> Date: 2016-11-28T19:47:42Z BEAM-1047 Update dataflow runner code to use DataflowClient wrapper. commit b2b570f27808b1671bf6cd0fc60f874da671d4ca Author: bchambers <bchambers@google.com> Date: 2016-12-07T01:08:13Z Closes #1434 commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152 Author: Sam McVeety <sgmc@google.com> Date: 2016-10-30T18:58:44Z BigQueryIO.Read: support runtime options commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-07T05:28:13Z FLINK-1102 Fix Aggregator Registration in Flink Batch Runner commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-07T07:26:02Z This closes #1530 commit baf5e6bd9b1011f4c5c3974aa46393471b340c15 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2016-12-07T07:37:33Z BEAM-1094 Set test scope for Kafka IO and junit commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f Author: Sela <ansela@paypal.com> Date: 2016-12-07T08:31:38Z This closes #1531 commit dce3a196a3a26fdd42225520faf3d9084ee48123 Author: Sela <ansela@paypal.com> Date: 2016-12-07T09:20:07Z BEAM-329 Update Spark runner README. commit 02bb8c375c48847b1686d70184fb194500a62e8c Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2016-12-07T11:51:09Z BEAM-329 This closes #1532 commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f Author: Kenneth Knowles <klk@google.com> Date: 2016-12-06T23:20:28Z Port most of DoFnRunner Javadoc to new DoFn commit 1526184ae8be1f8ae6863f830653204157a584cd Author: Thomas Groh <tgroh@google.com> Date: 2016-12-07T16:51:02Z This closes #1527 commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4 Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T05:01:37Z Port ParDoTest from OldDoFn to new DoFn commit ae52ec1bc3f3120e9f8e150e50c18564055a467c Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T17:00:18Z This closes #1529 commit 55d333bff68809ff1a9154491ace02d2d16e3b85 Author: Thomas Groh <tgroh@google.com> Date: 2016-12-05T22:29:05Z Only provide expanded Inputs and Outputs This removes PInput and POutput from the immediate API Surface of TransformHierarchy.Node, and forces Pipeline Visitors to access only the expanded version of the output. This is part of the move towards the runner-agnostic representation of a graph. commit 5b31a369962907e257de8019fbf6cde4c615b1c0 Author: Thomas Groh <tgroh@google.com> Date: 2016-12-07T17:14:38Z This closes #1511 commit 43fef2775145f67def3ab8a246ecca192a7d650b Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-07T12:06:57Z BEAM-905 Add shading config to examples archetype and enable it for Flink This makes the Flink quickstart work out of the box. commit b44a7ac4a2a0ac9fd7a8e38cc5eea5073c629085 Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-07T22:55:03Z Closes #1533 commit 6439f701d1008d6a0432828e11e0fcc8a4fe6ecc Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-07T23:40:58Z BEAM-1108 DataflowRunner: remove deprecated TEARDOWN_POLICY control commit 3b2e0290ddbedd199926a70f36c02ea4515841cb Author: Dan Halperin <dhalperi@google.com> Date: 2016-12-08T01:18:11Z Closes #1546 commit 1b1252074dd6b57f4fb88ceb82c704d3d3d8147f Author: Alexey Diomin <diominay@gmail.com> Date: 2016-12-07T05:39:27Z BEAM-1095 Add support set config for reuse-object on flink commit c53e0b1623e3ee3c08c329e2716440f031681591 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-08T01:44:07Z This closes #1518 commit f1a5704a505b01d7d4649b61d1f6697859367964 Author: Alexey Diomin <diominay@gmail.com> Date: 2016-12-07T05:48:35Z BEAM-1096 Flink streaming side output optimization using SplitStream commit 6807480a97f2315b3f48ad8dd5accb4e30475fa4 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2016-12-08T01:55:32Z This closes #1520 commit 3f8c80769a3bb38da64c6628fd8c4669fcac794b Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T04:10:06Z Add DoFn.OnTimerContext commit 2883062eebe8dba849ab89627f6aeb53266ac1a8 Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T04:10:21Z Access to OnTimerContext via DoFnInvokers.ArgumentProvider commit 42b506f06dbd73e03a2cfad4e7677e9698b3c020 Author: Kenneth Knowles <klk@google.com> Date: 2016-12-07T04:18:18Z Add OnTimerContext parameter support to DoFnSignature
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/incubator-beam/pull/1663

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1663
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/1828

          BEAM-79 Fix PostCommit test confs for Gearpump runner

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-post-commit

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/1828.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1828


          commit 4fd216b6b9f7427a7df118a60202752ebeb1836d
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-01-24T05:16:07Z

          BEAM-79 Fix PostCommit test confs for Gearpump runner


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/1828 BEAM-79 Fix PostCommit test confs for Gearpump runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-post-commit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/1828.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1828 commit 4fd216b6b9f7427a7df118a60202752ebeb1836d Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-01-24T05:16:07Z BEAM-79 Fix PostCommit test confs for Gearpump runner
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/1828

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/1828
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/1935

          BEAM-79 Fix Gearpump runner for WindowingTest

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-window

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/1935.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1935


          commit 82994256798fe85c4e2fbbab8262836c51ae1ab8
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-02-07T14:14:18Z

          BEAM-79 Fix Gearpump runner for WindowingTest


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/1935 BEAM-79 Fix Gearpump runner for WindowingTest Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-window Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/1935.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1935 commit 82994256798fe85c4e2fbbab8262836c51ae1ab8 Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-02-07T14:14:18Z BEAM-79 Fix Gearpump runner for WindowingTest
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/1935

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/1935
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/2150

          BEAM-79 Add SideInput support for GearpumpRunner

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-fix

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/2150.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2150


          commit 08b18bf326a7ac1bae485fbdf73aeab0de46c716
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-02-14T04:33:31Z

          BEAM-79 Add SideInput support for GearpumpRunner


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/2150 BEAM-79 Add SideInput support for GearpumpRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.txt ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2150 commit 08b18bf326a7ac1bae485fbdf73aeab0de46c716 Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-02-14T04:33:31Z BEAM-79 Add SideInput support for GearpumpRunner
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/2150

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/2150
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/2241

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/2241
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user huafengw opened a pull request:

          https://github.com/apache/beam/pull/2607

          BEAM-79 Upgrade Gearpump to a stable version

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
            Travis-CI on your fork and ensure the whole test matrix passes).
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/huafengw/beam upgrade

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/2607.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2607


          commit 3ba9b229a0a2cfcd84d5b83f208748c060caf20d
          Author: huafengw <fvunicorn@gmail.com>
          Date: 2017-04-20T01:36:45Z

          BEAM-79 Upgrade Gearpump to a stable version


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user huafengw opened a pull request: https://github.com/apache/beam/pull/2607 BEAM-79 Upgrade Gearpump to a stable version Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/huafengw/beam upgrade Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2607.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2607 commit 3ba9b229a0a2cfcd84d5b83f208748c060caf20d Author: huafengw <fvunicorn@gmail.com> Date: 2017-04-20T01:36:45Z BEAM-79 Upgrade Gearpump to a stable version
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user huafengw closed the pull request at:

          https://github.com/apache/beam/pull/2607

          Show
          githubbot ASF GitHub Bot added a comment - Github user huafengw closed the pull request at: https://github.com/apache/beam/pull/2607
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/2888

          BEAM-79 Merge master into gearpump-runner branch

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/incubator-beam sync-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/2888.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2888


          commit d9595d35176d2572423f6be069982b05eea204dd
          Author: Maria Garcia Herrero <mariagh@google.com>
          Date: 2017-04-26T23:22:09Z

          Add extra message to deprecated annotation

          commit 42e3a6f8588046f257e5c780dec12d16d0314a66
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-04-27T18:25:18Z

          This closes #2719

          commit 9174ebf48391f393761b2ce085d2f165ebb2c7bb
          Author: Tibor Kiss <tibor.kiss@gmail.com>
          Date: 2017-04-27T12:12:13Z

          BEAM-1695 Improve Python-SDK's programming guide

          commit 884935cb98f0e29bd965fda9182970423d111d7e
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-04-27T19:03:22Z

          This closes #2738

          commit e047b69efab9c988011303cf2eda86ac408b38c2
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T15:30:47Z

          BEAM-2096 Make DataflowMetrics more resilient

          DataflowMetrics seems to have many hard-coded assumptions about what will be
          returned by the Dataflow service that, which will likely break when users
          use new types of metrics or if the Dataflow service makes minor adjustments
          in our it sends metrics back to the user.

          In order for code to continue working in these cases, handle errors by logging
          and skipping.

          commit fdf2de9994268f5e1d704aa30a4504ac6ee624b8
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T20:10:32Z

          This closes #2739

          commit f93a27755921e73141fd97de042e6dcd04e10a47
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-26T00:29:02Z

          BEAM-59 Add FileSystems#matchNewResource

          The new FileSystems API needs a way to generate a ResourceId for a
          resource that does not exist. This does not come up in sources, because
          we typically are just matching existing files. However, sinks need the
          ability to reference a new directory (say, in order to create it).

          Couldn't think of anything better than a simple function that says
          "treat this string as a full resource path with the specified type",
          which is what FileSystems#matchNewResource is.

          commit 14a90d7c5196d96de66e897687d28202d790439f
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T21:56:24Z

          This closes #2728

          commit d3b399fa84bff45b0960368d32d4c69e433f4c94
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-04-26T22:12:54Z

          Remove Coder Encoding IDs

          These were never safe to use, and will not be used once Coders have been
          migrated to the Runner API representation.

          commit c4936958834aa55f0f1fcb73fe9b8419c4c9eb9b
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-04-27T22:22:55Z

          This closes #2717

          commit 1925a50d860b7b8f8422f1c2f251d0ea11def736
          Author: Aviem Zur <aviemzur@gmail.com>
          Date: 2017-04-27T20:48:02Z

          BEAM-2029 NullPointerException when using multi output ParDo in Spark runner in streaming mode.

          commit a46eb1a80074dc9c382da4d71725ffc61f2e65ba
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T22:33:39Z

          This closes #2746

          commit 9a3b830c8737859c6afaa6e6e1922b037a78741f
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T22:35:10Z

          BEAM-59 LocalResources: helper utility to create LocalResourceId

          Primarily for testing.

          commit f71395bfd8f94cbe8d78098f5bb66a1dce65e74b
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T23:30:50Z

          This closes #2751

          commit 95fdb8f4dbb6bb8ec6d80152aa85cc970ab13366
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-27T20:51:53Z

          WriteFiles: trivial message fix

          commit 674ae9a475f1f236230697d5cb1c33d36fec49db
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-28T00:20:54Z

          This closes #2755

          commit 06f4ef40bec7d6d3c5826fb9e0c164eb9c05efe7
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-04-28T00:14:41Z

          Add a simple snippet for a composite transform.

          commit 948f7ef2fc37b30809cf1adb2631eeb6a1e5eff3
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-04-28T00:35:37Z

          This closes #2756

          commit e1d5174eaffef3b2ac7df6187c43ffbccce32b09
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-04-27T23:24:57Z

          BEAM-2031 Add support to PipelineOptionsFactory/ProxyInvocationHandler for auto registration of Jackson modules

          This is towards adding a mixin module for Hadoop Configuration so that it can live on a PipelineOptions object.

          commit 185dc47982078124abf6e851ad1c66ebc4ae8c54
          Author: Lukasz Cwik <lcwik@google.com>
          Date: 2017-04-28T03:15:40Z

          BEAM-2031 Add support to PipelineOptionsFactory/ProxyInvocationHandler for auto registration of Jackson modules

          This closes #2752

          commit 34c3ee7bfe0e0b6b95d0e9ec421a061116fc19b0
          Author: Vikas Kedigehalli <vikasrk@google.com>
          Date: 2017-04-27T21:58:24Z

          BEAM-59 Remove IOChannelFactory usage from NumberedShardedFile/ExplicitShardedFile

          commit 47aaf11250e61d61c0bb3616063dbd5b7f2adb88
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-04-28T06:03:12Z

          This closes #2749

          commit f37d954873ae03cdae854837570b62e334be9022
          Author: Thomas Weise <thw@apache.org>
          Date: 2017-04-27T09:06:50Z

          BEAM-1981 Fix ApexTimerInternals serialization error.

          commit 40a924d8d4e06ecbd7252ddbd6558415f899c123
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-04-27T18:34:01Z

          BEAM-1871 Remove irrelevant runtime dependencies.

          Confirmed that sdks/java/io/gcp uses 1.22 for the removed libraries.

          commit 849e39dfcf28a6edbf3f4c0469229778e0a52cb3
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-04-28T15:14:46Z

          BEAM-1871 Remove irrelevant runtime dependencies

          This closes #2748

          commit ce88c88b14e963ac17fac83dd19495c835c1f6cb
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-04-28T01:13:43Z

          BEAM-2005 Move getScheme from FileSystemRegistrar to FileSystem

          Note that I needed to update FileSystems to instantiate the FileSystem(s) upfront instead of remembering the mapping from scheme to registrar.

          commit ddabab3faded59e782455486fedf74b2ff60b60c
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-04-28T16:00:26Z

          BEAM-2005 Move getScheme from FileSystemRegistrar to FileSystem

          This closes #2757

          commit fa422b9a81c303616786e8198d22c70df983d28e
          Author: Ismaël Mejía <iemejia@apache.org>
          Date: 2017-04-28T15:25:30Z

          Remove missing references to the flink examples package after refactor

          commit d08b72892bf900995667a37239e610bc2cd960d0
          Author: Ismaël Mejía <iemejia@apache.org>
          Date: 2017-04-28T16:26:08Z

          This closes #2760

          commit 594705d717dd6e5b81c08dcf4c56f8f6e7422737
          Author: Peter Gergo Barna <pbarna@hortonworks.com>
          Date: 2017-04-27T12:37:26Z

          BEAM-2094 WordCount examples produce garbage for non-English input text


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/2888 BEAM-79 Merge master into gearpump-runner branch Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-beam sync-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2888.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2888 commit d9595d35176d2572423f6be069982b05eea204dd Author: Maria Garcia Herrero <mariagh@google.com> Date: 2017-04-26T23:22:09Z Add extra message to deprecated annotation commit 42e3a6f8588046f257e5c780dec12d16d0314a66 Author: Ahmet Altay <altay@google.com> Date: 2017-04-27T18:25:18Z This closes #2719 commit 9174ebf48391f393761b2ce085d2f165ebb2c7bb Author: Tibor Kiss <tibor.kiss@gmail.com> Date: 2017-04-27T12:12:13Z BEAM-1695 Improve Python-SDK's programming guide commit 884935cb98f0e29bd965fda9182970423d111d7e Author: Ahmet Altay <altay@google.com> Date: 2017-04-27T19:03:22Z This closes #2738 commit e047b69efab9c988011303cf2eda86ac408b38c2 Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T15:30:47Z BEAM-2096 Make DataflowMetrics more resilient DataflowMetrics seems to have many hard-coded assumptions about what will be returned by the Dataflow service that, which will likely break when users use new types of metrics or if the Dataflow service makes minor adjustments in our it sends metrics back to the user. In order for code to continue working in these cases, handle errors by logging and skipping. commit fdf2de9994268f5e1d704aa30a4504ac6ee624b8 Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T20:10:32Z This closes #2739 commit f93a27755921e73141fd97de042e6dcd04e10a47 Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-26T00:29:02Z BEAM-59 Add FileSystems#matchNewResource The new FileSystems API needs a way to generate a ResourceId for a resource that does not exist. This does not come up in sources, because we typically are just matching existing files. However, sinks need the ability to reference a new directory (say, in order to create it). Couldn't think of anything better than a simple function that says "treat this string as a full resource path with the specified type", which is what FileSystems#matchNewResource is. commit 14a90d7c5196d96de66e897687d28202d790439f Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T21:56:24Z This closes #2728 commit d3b399fa84bff45b0960368d32d4c69e433f4c94 Author: Thomas Groh <tgroh@google.com> Date: 2017-04-26T22:12:54Z Remove Coder Encoding IDs These were never safe to use, and will not be used once Coders have been migrated to the Runner API representation. commit c4936958834aa55f0f1fcb73fe9b8419c4c9eb9b Author: Thomas Groh <tgroh@google.com> Date: 2017-04-27T22:22:55Z This closes #2717 commit 1925a50d860b7b8f8422f1c2f251d0ea11def736 Author: Aviem Zur <aviemzur@gmail.com> Date: 2017-04-27T20:48:02Z BEAM-2029 NullPointerException when using multi output ParDo in Spark runner in streaming mode. commit a46eb1a80074dc9c382da4d71725ffc61f2e65ba Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T22:33:39Z This closes #2746 commit 9a3b830c8737859c6afaa6e6e1922b037a78741f Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T22:35:10Z BEAM-59 LocalResources: helper utility to create LocalResourceId Primarily for testing. commit f71395bfd8f94cbe8d78098f5bb66a1dce65e74b Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T23:30:50Z This closes #2751 commit 95fdb8f4dbb6bb8ec6d80152aa85cc970ab13366 Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-27T20:51:53Z WriteFiles: trivial message fix commit 674ae9a475f1f236230697d5cb1c33d36fec49db Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-28T00:20:54Z This closes #2755 commit 06f4ef40bec7d6d3c5826fb9e0c164eb9c05efe7 Author: Ahmet Altay <altay@google.com> Date: 2017-04-28T00:14:41Z Add a simple snippet for a composite transform. commit 948f7ef2fc37b30809cf1adb2631eeb6a1e5eff3 Author: Ahmet Altay <altay@google.com> Date: 2017-04-28T00:35:37Z This closes #2756 commit e1d5174eaffef3b2ac7df6187c43ffbccce32b09 Author: Luke Cwik <lcwik@google.com> Date: 2017-04-27T23:24:57Z BEAM-2031 Add support to PipelineOptionsFactory/ProxyInvocationHandler for auto registration of Jackson modules This is towards adding a mixin module for Hadoop Configuration so that it can live on a PipelineOptions object. commit 185dc47982078124abf6e851ad1c66ebc4ae8c54 Author: Lukasz Cwik <lcwik@google.com> Date: 2017-04-28T03:15:40Z BEAM-2031 Add support to PipelineOptionsFactory/ProxyInvocationHandler for auto registration of Jackson modules This closes #2752 commit 34c3ee7bfe0e0b6b95d0e9ec421a061116fc19b0 Author: Vikas Kedigehalli <vikasrk@google.com> Date: 2017-04-27T21:58:24Z BEAM-59 Remove IOChannelFactory usage from NumberedShardedFile/ExplicitShardedFile commit 47aaf11250e61d61c0bb3616063dbd5b7f2adb88 Author: Dan Halperin <dhalperi@google.com> Date: 2017-04-28T06:03:12Z This closes #2749 commit f37d954873ae03cdae854837570b62e334be9022 Author: Thomas Weise <thw@apache.org> Date: 2017-04-27T09:06:50Z BEAM-1981 Fix ApexTimerInternals serialization error. commit 40a924d8d4e06ecbd7252ddbd6558415f899c123 Author: Luke Cwik <lcwik@google.com> Date: 2017-04-27T18:34:01Z BEAM-1871 Remove irrelevant runtime dependencies. Confirmed that sdks/java/io/gcp uses 1.22 for the removed libraries. commit 849e39dfcf28a6edbf3f4c0469229778e0a52cb3 Author: Luke Cwik <lcwik@google.com> Date: 2017-04-28T15:14:46Z BEAM-1871 Remove irrelevant runtime dependencies This closes #2748 commit ce88c88b14e963ac17fac83dd19495c835c1f6cb Author: Luke Cwik <lcwik@google.com> Date: 2017-04-28T01:13:43Z BEAM-2005 Move getScheme from FileSystemRegistrar to FileSystem Note that I needed to update FileSystems to instantiate the FileSystem(s) upfront instead of remembering the mapping from scheme to registrar. commit ddabab3faded59e782455486fedf74b2ff60b60c Author: Luke Cwik <lcwik@google.com> Date: 2017-04-28T16:00:26Z BEAM-2005 Move getScheme from FileSystemRegistrar to FileSystem This closes #2757 commit fa422b9a81c303616786e8198d22c70df983d28e Author: Ismaël Mejía <iemejia@apache.org> Date: 2017-04-28T15:25:30Z Remove missing references to the flink examples package after refactor commit d08b72892bf900995667a37239e610bc2cd960d0 Author: Ismaël Mejía <iemejia@apache.org> Date: 2017-04-28T16:26:08Z This closes #2760 commit 594705d717dd6e5b81c08dcf4c56f8f6e7422737 Author: Peter Gergo Barna <pbarna@hortonworks.com> Date: 2017-04-27T12:37:26Z BEAM-2094 WordCount examples produce garbage for non-English input text
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/2888

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/2888
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3172

          BEAM-79 Sync gearpump-runner with master

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam sync-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3172.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3172


          commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-06T00:16:34Z

          BEAM-2190 pom.xml: do a better job of dependency management

          Even if Beam appears to have the correct dependencies, we cannot
          guarantee that modules that depend on us transitively get the right
          dependencies. For example, even though grpc-protobuf-lite has
          protobuf-lite excluded, and the Maven Enforcer banned-dependencies
          check passes... if a user happens to get a transitive dependency on
          grpc-all first, they may pull in grpc-protobuf from that other source
          without the exclusion. Thus we need to exclude protobuf-lite from
          grpc-all as well.

          While we're here, also add guava-jdk5 to the set of banned dependencies,
          though (as above) we cannot currently properly identify the places it
          might be transitively exposed in a users' pom.xml.

          commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-08T18:35:56Z

          This closes #2934

          commit 5bac40efcc7bcd9d53183579ec6089c23aeda782
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-06T02:06:03Z

          BEAM-2212 FileBasedSource: improve message when logging.

          ValueProvider should not be printed, rather the string instead.

          commit 9423babd8f827e843723c218441e9a91aaa7b361
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-08T16:59:16Z

          BEAM-2212 FileBasedSource: refactor to remove uses of fileOrPatternSpec.get()

          Makes it less likely to have errors from printing ValueProviders instead of runtime values

          commit e1791c3f83a007013625b43f9976e7ffa64e8a49
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-08T18:48:42Z

          This closes #2956

          commit 13e51c9cd114eec73c47b71f46214dd92ee81048
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-06T02:49:29Z

          Convert Executor Services to use Daemon Threads

          This will cause the DirectRunner to automatically shut down when the
          worker threads are shut down.

          commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-05-08T19:52:09Z

          This closes #2954

          commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80
          Author: Sourabh Bajaj <sourabhbajaj@google.com>
          Date: 2017-05-07T20:55:20Z

          BEAM-2206 Move pipelineOptions into options modules

          commit 604c2f59e2a928a95106814b0b55b8097dc39faa
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-08T19:56:15Z

          This closes #2925

          commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18
          Author: Reuven Lax <relax@google.com>
          Date: 2017-05-08T16:06:55Z

          TableDescription is allowed to be null.

          commit c481396edc7a3e906cd79dcc2a356baf9c93895e
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-08T20:41:01Z

          This closes #2953

          commit 1fd15bd7441e1be72feb3f9535ffe536d730a531
          Author: Reuven Lax <relax@google.com>
          Date: 2017-05-07T18:50:08Z

          Shortens PubsubIO names for reading/writing messages.

          commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-08T20:45:25Z

          This closes #2950

          commit b7635aadc2532b5c3bdc38f070865091ebf901c1
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-08T20:42:02Z

          Remove apex README.md. This information is already in the website in
          quickstart and apex runner pages.
          https://github.com/apache/beam-site/pull/232 moves small bits of missing
          content.

          commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-08T21:28:36Z

          This closes #2962

          commit fba4c010ce90db8b5cdac489d47126ca3ee8db65
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-08T22:00:33Z

          Remove readme.md files that are covered in web site

          commit 152c5bcc7f6fb53565ed132653384f4244542b59
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-08T22:04:46Z

          This closes #2966

          commit 78c29238aa471138fb55681d8700b095a3606e0d
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-05-08T21:54:17Z

          BEAM-2224 Fix temp file management on Windows.

          commit f315cc9ff649ccee9c9aff18f4af30cfbd58e5e5
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-05-08T23:49:05Z

          Closes #2965

          commit fd1dda09fc4db48cfbc21afd6310320533fffe78
          Author: Reuven Lax <relax@google.com>
          Date: 2017-04-29T14:33:54Z

          BEAM-2154 More scalable dynamic BigQueryIO.Write

          If too many tables are generated in a bundle, spill and group the results
          before writing files. Generating hundreds or thousands of file write buffers
          in a single bundle was causing workers to crash with out of memory.

          commit 6a2586a4ee279f66fd4eae816e93fa8e397acf6b
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-09T01:02:14Z

          This closes #2883

          commit 367fcb28d544934797d25cb34d54136b2d7d6e99
          Author: Vikas Kedigehalli <vikasrk@google.com>
          Date: 2017-05-06T02:24:51Z

          Remove google api BackOff usage from sdks/core

          commit 87578a6a69f15bca502d9b4c15999aa383e539a5
          Author: Vikas Kedigehalli <vikasrk@google.com>
          Date: 2017-05-06T02:33:45Z

          Use guava Base64 encoding instead of google api client

          commit 88e044fda0a7d0ea1372f075a4c7a4bd54af47da
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-09T02:20:01Z

          BEAM-1871 Remove Google API client dependency from sdks/core

          This closes #2937

          commit 16f355f7e481ebf029c0edb878742f6fea57b6cd
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-08T22:37:29Z

          BEAM-2211 DataflowRunner: remove validation of file read/write paths

          Now that users can implement and register custom FileSystems,
          we can no longer really effectively validate filesystems they
          can read or write files from. They can even register file://
          to point to some HDFS path, e.g.,

          commit 23731fe7ab12ea5d3ebb7b3f16c788b83d0caf0b
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-09T03:13:38Z

          This closes #2968

          commit 27e9a060ed593b2b53b88481591f14b1a274c61b
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-05-05T23:20:37Z

          Remove explicit used of nested contexts.

          find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), *context.nested..[)]/\1)/'
          find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), *nestedContext[)]/\1)/'
          find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), *Context.NESTED[)]/\1)/'
          find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), [^ ].Context.NESTED[)]/\1)/'

          Added back explicit context in CoGbkResult.java due to compile error.

          commit 996dce37b76b103f104328b7caa65f73a1bcb15a
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-05-05T23:36:47Z

          Remove contexts from coders where they'll never be used.

          commit b7f3341ed4fc073a834a55773bcb4a2c7f821c52
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-05-06T00:24:02Z

          automated context removal or redirection

          commit 45e09b2df7a97cfbbe0d8013c16cfdd7a55afbde
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-05-06T00:27:13Z

          get it compiling


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3172 BEAM-79 Sync gearpump-runner with master Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam sync-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3172.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3172 commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-06T00:16:34Z BEAM-2190 pom.xml: do a better job of dependency management Even if Beam appears to have the correct dependencies, we cannot guarantee that modules that depend on us transitively get the right dependencies. For example, even though grpc-protobuf-lite has protobuf-lite excluded, and the Maven Enforcer banned-dependencies check passes... if a user happens to get a transitive dependency on grpc-all first, they may pull in grpc-protobuf from that other source without the exclusion. Thus we need to exclude protobuf-lite from grpc-all as well. While we're here, also add guava-jdk5 to the set of banned dependencies, though (as above) we cannot currently properly identify the places it might be transitively exposed in a users' pom.xml. commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-08T18:35:56Z This closes #2934 commit 5bac40efcc7bcd9d53183579ec6089c23aeda782 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-06T02:06:03Z BEAM-2212 FileBasedSource: improve message when logging. ValueProvider should not be printed, rather the string instead. commit 9423babd8f827e843723c218441e9a91aaa7b361 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-08T16:59:16Z BEAM-2212 FileBasedSource: refactor to remove uses of fileOrPatternSpec.get() Makes it less likely to have errors from printing ValueProviders instead of runtime values commit e1791c3f83a007013625b43f9976e7ffa64e8a49 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-08T18:48:42Z This closes #2956 commit 13e51c9cd114eec73c47b71f46214dd92ee81048 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-06T02:49:29Z Convert Executor Services to use Daemon Threads This will cause the DirectRunner to automatically shut down when the worker threads are shut down. commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c Author: Thomas Groh <tgroh@google.com> Date: 2017-05-08T19:52:09Z This closes #2954 commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80 Author: Sourabh Bajaj <sourabhbajaj@google.com> Date: 2017-05-07T20:55:20Z BEAM-2206 Move pipelineOptions into options modules commit 604c2f59e2a928a95106814b0b55b8097dc39faa Author: Ahmet Altay <altay@google.com> Date: 2017-05-08T19:56:15Z This closes #2925 commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18 Author: Reuven Lax <relax@google.com> Date: 2017-05-08T16:06:55Z TableDescription is allowed to be null. commit c481396edc7a3e906cd79dcc2a356baf9c93895e Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-08T20:41:01Z This closes #2953 commit 1fd15bd7441e1be72feb3f9535ffe536d730a531 Author: Reuven Lax <relax@google.com> Date: 2017-05-07T18:50:08Z Shortens PubsubIO names for reading/writing messages. commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-08T20:45:25Z This closes #2950 commit b7635aadc2532b5c3bdc38f070865091ebf901c1 Author: Ahmet Altay <altay@google.com> Date: 2017-05-08T20:42:02Z Remove apex README.md. This information is already in the website in quickstart and apex runner pages. https://github.com/apache/beam-site/pull/232 moves small bits of missing content. commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60 Author: Ahmet Altay <altay@google.com> Date: 2017-05-08T21:28:36Z This closes #2962 commit fba4c010ce90db8b5cdac489d47126ca3ee8db65 Author: Ahmet Altay <altay@google.com> Date: 2017-05-08T22:00:33Z Remove readme.md files that are covered in web site commit 152c5bcc7f6fb53565ed132653384f4244542b59 Author: Ahmet Altay <altay@google.com> Date: 2017-05-08T22:04:46Z This closes #2966 commit 78c29238aa471138fb55681d8700b095a3606e0d Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-05-08T21:54:17Z BEAM-2224 Fix temp file management on Windows. commit f315cc9ff649ccee9c9aff18f4af30cfbd58e5e5 Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-05-08T23:49:05Z Closes #2965 commit fd1dda09fc4db48cfbc21afd6310320533fffe78 Author: Reuven Lax <relax@google.com> Date: 2017-04-29T14:33:54Z BEAM-2154 More scalable dynamic BigQueryIO.Write If too many tables are generated in a bundle, spill and group the results before writing files. Generating hundreds or thousands of file write buffers in a single bundle was causing workers to crash with out of memory. commit 6a2586a4ee279f66fd4eae816e93fa8e397acf6b Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-09T01:02:14Z This closes #2883 commit 367fcb28d544934797d25cb34d54136b2d7d6e99 Author: Vikas Kedigehalli <vikasrk@google.com> Date: 2017-05-06T02:24:51Z Remove google api BackOff usage from sdks/core commit 87578a6a69f15bca502d9b4c15999aa383e539a5 Author: Vikas Kedigehalli <vikasrk@google.com> Date: 2017-05-06T02:33:45Z Use guava Base64 encoding instead of google api client commit 88e044fda0a7d0ea1372f075a4c7a4bd54af47da Author: Luke Cwik <lcwik@google.com> Date: 2017-05-09T02:20:01Z BEAM-1871 Remove Google API client dependency from sdks/core This closes #2937 commit 16f355f7e481ebf029c0edb878742f6fea57b6cd Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-08T22:37:29Z BEAM-2211 DataflowRunner: remove validation of file read/write paths Now that users can implement and register custom FileSystems, we can no longer really effectively validate filesystems they can read or write files from. They can even register file:// to point to some HDFS path, e.g., commit 23731fe7ab12ea5d3ebb7b3f16c788b83d0caf0b Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-09T03:13:38Z This closes #2968 commit 27e9a060ed593b2b53b88481591f14b1a274c61b Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-05-05T23:20:37Z Remove explicit used of nested contexts. find . -type f -name ' .java' | xargs sed -i '' 's/( [.] ..code [(] . ), *context.nested.. [)] /\1)/' find . -type f -name ' .java' | xargs sed -i '' 's/( [.] ..code [(] . ), *nestedContext [)] /\1)/' find . -type f -name ' .java' | xargs sed -i '' 's/( [.] ..code [(] . ), *Context.NESTED [)] /\1)/' find . -type f -name ' .java' | xargs sed -i '' 's/( [.] ..code [(] . ), [^ ] .Context.NESTED [)] /\1)/' Added back explicit context in CoGbkResult.java due to compile error. commit 996dce37b76b103f104328b7caa65f73a1bcb15a Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-05-05T23:36:47Z Remove contexts from coders where they'll never be used. commit b7f3341ed4fc073a834a55773bcb4a2c7f821c52 Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-05-06T00:24:02Z automated context removal or redirection commit 45e09b2df7a97cfbbe0d8013c16cfdd7a55afbde Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-05-06T00:27:13Z get it compiling
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3172

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3172
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3181

          BEAM-79 Respect WindowFn#getOutputTime in gearpump-runner

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam fix_windowing_test

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3181.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3181


          commit 7328ff21f2b805801123c0458d4fad22563ad384
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-05-19T01:19:42Z

          BEAM-79 Respect WindowFn#getOutputTime in gearpump-runner


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3181 BEAM-79 Respect WindowFn#getOutputTime in gearpump-runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam fix_windowing_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3181.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3181 commit 7328ff21f2b805801123c0458d4fad22563ad384 Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-05-19T01:19:42Z BEAM-79 Respect WindowFn#getOutputTime in gearpump-runner
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3292

          BEAM-79 Merge master into gearpump-runner

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam sync-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3292.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3292


          commit 66e564e507b1ad5b9e379e043a193b556e1f3e76
          Author: Borisa Zivkovic <borisa.zivkovic@huawei.com>
          Date: 2017-05-15T08:24:10Z

          BEAM-1702 Document how to use BigtableIO with local emulator

          commit 8ef812defc5237934f4f3efa6bd8c4c4240cd7d0
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-18T20:07:13Z

          This closes #3141

          commit a8d2125e5783a556056e88dad8fe3c0a397920d5
          Author: huafengw <fvunicorn@gmail.com>
          Date: 2017-05-09T03:21:44Z

          BEAM-659 WindowFn#isCompatible should provide a meaningful reason

          commit 8c572ef0b998a4fa529e2f4ba0e622b3db9f4ec6
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-05-18T20:27:59Z

          This closes #2985: WindowFn#isCompatible should provide a meaningful reason

          commit 26a615e885b39f92dc81d383c8115906c280187a
          Author: Andrew Martin <andrewsmartin.mg@gmail.com>
          Date: 2017-05-18T19:52:09Z

          Remove incorrect `Message` bound on writeAvros

          commit 6a7eeeb932ffb33c876c0d11d7f325c47606d8de
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-18T20:52:33Z

          This closes #3176: Remove incorrect `Message` bound on writeAvros

          commit 790e7fe6653b926044d3dfecdccbc2fda9c998f0
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-03-21T22:06:58Z

          Add ParDos

          Add ParDoPayloadTranslator to PTransformTranslator

          commit 2e0cf008303ff5ed48268550b9fe3cd1aa748c15
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-05-18T22:41:17Z

          This closes #2597

          commit 7a2e829c849d3828353994b8eddca4784a900aca
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-15T22:54:03Z

          Splits SplittableParDo into a core-construction part and a KWI-related part

          commit c93b3aa3d2081558a776fa12c564efb5f15090ab
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-18T23:15:00Z

          This closes #3156

          commit 54808be14ff2e86088807adc3c0dc0965c8a6f03
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-18T22:43:31Z

          BEAM-2318 Ensure that tables that don't exist are not created

          We use the same test table name in two tests: one that expects the table not to exist, another that
          creates the table. Obviously, the correctness of this will depend on the order in which unit tests
          are executed.

          Fix the flake by using different and more explicit test names.

          commit f458065dad0429ba2639510ea460cf7f2679ed36
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-18T23:41:41Z

          This closes #3179

          commit 1da8da79dcc623c53b35d97419566b736447f6b6
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2017-05-18T20:19:29Z

          BEAM-2391 Clone Scan in HBaseReader

          commit 49245080ad6393b5678077caa047f7a6ef0efffc
          Author: Ismaël Mejía <iemejia@apache.org>
          Date: 2017-05-19T08:50:06Z

          This closes #3178

          commit f969ea443c90345cf8c33f953c399e46223e8d03
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-15T20:39:12Z

          BEAM-2252 Use pluginManagement instead of redefining maven-shade-plugin to inherit build order from root pom.xml

          commit ea5b9dabbfb55ff47a4f5983a8cbf04b0949c77c
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-19T17:37:10Z

          BEAM-2252 Use pluginManagement instead of redefining maven-shade-plugin to inherit build order from root pom.xml

          This closes #3152

          commit 3efb34b8322ac5933d9d71255d67e99d8e8361d7
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-19T02:00:00Z

          update dataflow python container

          commit a58275a2f904ab1304de7ca5295f27a4f6df6f1d
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-05-19T17:43:15Z

          This closes #3182

          commit bac7bde5f878e6b2104c2e7cf3d01894a8f1b71d
          Author: Mark Liu <markliu@google.com>
          Date: 2017-05-17T21:24:25Z

          BEAM-2164 Fix generate-sources.sh if directory contains space

          commit ce00d2466a9495f4eb262b5f7d2afec5788fcfe2
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-05-19T17:44:54Z

          This closes #3171: BEAM-2164 Fix generate-sources.sh if directory contains space

          commit c7bd2b43ddf726248291a653f8d7d796a51800a0
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-05-19T18:40:20Z

          Move snappy-java version to dependencyManagement

          commit c4d14f8be7bb72fd653f1ab8e8080fc2b65f6672
          Author: Colin Phipps <cph@moria.org.uk>
          Date: 2017-04-25T15:28:28Z

          Comply with byte limit for Datastore Commit.

          commit dcf405644e2b51303e1d2c12592fe82ee01eb32f
          Author: Colin Phipps <fipsy@google.com>
          Date: 2017-05-09T09:40:50Z

          End-to-end test for large entity writes.

          commit de95c7f5bf3b8391486b898b0ffb1a0a05338725
          Author: Colin Phipps <fipsy@google.com>
          Date: 2017-05-09T15:07:51Z

          Make the large entities test neater.

          commit 9a6baefcd8668b145e86140f64b9d18c1c21890c
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-05-19T20:11:50Z

          This closes #2948: BEAM-991 Comply with byte limit for Datastore Commit.

          commit 383329d6c35e50c8d907ac97522a64236bb98dd0
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-12T23:44:10Z

          BEAM-1345 Remove FileSystems.setDefaultConfigInWorkers since Dataflow no longer depends on this

          commit 086148fa9bbdd53fb9684cbbf37346df5a0ec4df
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-19T20:29:31Z

          BEAM-1345 Remove FileSystems.setDefaultConfigInWorkers since Dataflow no longer depends on this

          This closes #3134

          commit 6c6d7877620af5265ed5ca14dd2ed7cd905b7404
          Author: wtanaka.com <wtanaka@yahoo.com>
          Date: 2017-05-19T06:42:46Z

          BEAM-2293 Rename Top.

          {Largest,Smallest=>Natural,Reversed}

          - [x] Make sure the PR title is formatted like:
          `BEAM-<Jira issue #> Description of pull request`
          - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
          Travis-CI on your fork and ensure the whole test matrix passes).
          - [x] Replace `<Jira issue #>` in the title with the actual Jira issue
          number, if there is one.
          - [x] If this contribution is large, please file an Apache
          [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).



          commit 64cea06b1c1cbb1413b0ec570d675975bbbc20ba
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-19T21:11:02Z

          BEAM-2293 Rename Top.{Largest,Smallest=>Natural,Reversed}

          This closes #3139

          commit 2ed68f7f49a6f84921b0de03a57b0bbe61d263a7
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-05-15T22:43:00Z

          BEAM-2253 Use resource transformer plugin to use versions defined in root pom.xml within maven archetype poms


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3292 BEAM-79 Merge master into gearpump-runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam sync-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3292 commit 66e564e507b1ad5b9e379e043a193b556e1f3e76 Author: Borisa Zivkovic <borisa.zivkovic@huawei.com> Date: 2017-05-15T08:24:10Z BEAM-1702 Document how to use BigtableIO with local emulator commit 8ef812defc5237934f4f3efa6bd8c4c4240cd7d0 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-18T20:07:13Z This closes #3141 commit a8d2125e5783a556056e88dad8fe3c0a397920d5 Author: huafengw <fvunicorn@gmail.com> Date: 2017-05-09T03:21:44Z BEAM-659 WindowFn#isCompatible should provide a meaningful reason commit 8c572ef0b998a4fa529e2f4ba0e622b3db9f4ec6 Author: Kenneth Knowles <klk@google.com> Date: 2017-05-18T20:27:59Z This closes #2985: WindowFn#isCompatible should provide a meaningful reason commit 26a615e885b39f92dc81d383c8115906c280187a Author: Andrew Martin <andrewsmartin.mg@gmail.com> Date: 2017-05-18T19:52:09Z Remove incorrect `Message` bound on writeAvros commit 6a7eeeb932ffb33c876c0d11d7f325c47606d8de Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-18T20:52:33Z This closes #3176: Remove incorrect `Message` bound on writeAvros commit 790e7fe6653b926044d3dfecdccbc2fda9c998f0 Author: Thomas Groh <tgroh@google.com> Date: 2017-03-21T22:06:58Z Add ParDos Add ParDoPayloadTranslator to PTransformTranslator commit 2e0cf008303ff5ed48268550b9fe3cd1aa748c15 Author: Thomas Groh <tgroh@google.com> Date: 2017-05-18T22:41:17Z This closes #2597 commit 7a2e829c849d3828353994b8eddca4784a900aca Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-15T22:54:03Z Splits SplittableParDo into a core-construction part and a KWI-related part commit c93b3aa3d2081558a776fa12c564efb5f15090ab Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-18T23:15:00Z This closes #3156 commit 54808be14ff2e86088807adc3c0dc0965c8a6f03 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-18T22:43:31Z BEAM-2318 Ensure that tables that don't exist are not created We use the same test table name in two tests: one that expects the table not to exist, another that creates the table. Obviously, the correctness of this will depend on the order in which unit tests are executed. Fix the flake by using different and more explicit test names. commit f458065dad0429ba2639510ea460cf7f2679ed36 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-18T23:41:41Z This closes #3179 commit 1da8da79dcc623c53b35d97419566b736447f6b6 Author: Dan Halperin <dhalperi@google.com> Date: 2017-05-18T20:19:29Z BEAM-2391 Clone Scan in HBaseReader commit 49245080ad6393b5678077caa047f7a6ef0efffc Author: Ismaël Mejía <iemejia@apache.org> Date: 2017-05-19T08:50:06Z This closes #3178 commit f969ea443c90345cf8c33f953c399e46223e8d03 Author: Luke Cwik <lcwik@google.com> Date: 2017-05-15T20:39:12Z BEAM-2252 Use pluginManagement instead of redefining maven-shade-plugin to inherit build order from root pom.xml commit ea5b9dabbfb55ff47a4f5983a8cbf04b0949c77c Author: Luke Cwik <lcwik@google.com> Date: 2017-05-19T17:37:10Z BEAM-2252 Use pluginManagement instead of redefining maven-shade-plugin to inherit build order from root pom.xml This closes #3152 commit 3efb34b8322ac5933d9d71255d67e99d8e8361d7 Author: Ahmet Altay <altay@google.com> Date: 2017-05-19T02:00:00Z update dataflow python container commit a58275a2f904ab1304de7ca5295f27a4f6df6f1d Author: Ahmet Altay <altay@google.com> Date: 2017-05-19T17:43:15Z This closes #3182 commit bac7bde5f878e6b2104c2e7cf3d01894a8f1b71d Author: Mark Liu <markliu@google.com> Date: 2017-05-17T21:24:25Z BEAM-2164 Fix generate-sources.sh if directory contains space commit ce00d2466a9495f4eb262b5f7d2afec5788fcfe2 Author: Kenneth Knowles <klk@google.com> Date: 2017-05-19T17:44:54Z This closes #3171: BEAM-2164 Fix generate-sources.sh if directory contains space commit c7bd2b43ddf726248291a653f8d7d796a51800a0 Author: Kenneth Knowles <klk@google.com> Date: 2017-05-19T18:40:20Z Move snappy-java version to dependencyManagement commit c4d14f8be7bb72fd653f1ab8e8080fc2b65f6672 Author: Colin Phipps <cph@moria.org.uk> Date: 2017-04-25T15:28:28Z Comply with byte limit for Datastore Commit. commit dcf405644e2b51303e1d2c12592fe82ee01eb32f Author: Colin Phipps <fipsy@google.com> Date: 2017-05-09T09:40:50Z End-to-end test for large entity writes. commit de95c7f5bf3b8391486b898b0ffb1a0a05338725 Author: Colin Phipps <fipsy@google.com> Date: 2017-05-09T15:07:51Z Make the large entities test neater. commit 9a6baefcd8668b145e86140f64b9d18c1c21890c Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-05-19T20:11:50Z This closes #2948: BEAM-991 Comply with byte limit for Datastore Commit. commit 383329d6c35e50c8d907ac97522a64236bb98dd0 Author: Luke Cwik <lcwik@google.com> Date: 2017-05-12T23:44:10Z BEAM-1345 Remove FileSystems.setDefaultConfigInWorkers since Dataflow no longer depends on this commit 086148fa9bbdd53fb9684cbbf37346df5a0ec4df Author: Luke Cwik <lcwik@google.com> Date: 2017-05-19T20:29:31Z BEAM-1345 Remove FileSystems.setDefaultConfigInWorkers since Dataflow no longer depends on this This closes #3134 commit 6c6d7877620af5265ed5ca14dd2ed7cd905b7404 Author: wtanaka.com <wtanaka@yahoo.com> Date: 2017-05-19T06:42:46Z BEAM-2293 Rename Top. {Largest,Smallest=>Natural,Reversed} - [x] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — commit 64cea06b1c1cbb1413b0ec570d675975bbbc20ba Author: Luke Cwik <lcwik@google.com> Date: 2017-05-19T21:11:02Z BEAM-2293 Rename Top.{Largest,Smallest=>Natural,Reversed} This closes #3139 commit 2ed68f7f49a6f84921b0de03a57b0bbe61d263a7 Author: Luke Cwik <lcwik@google.com> Date: 2017-05-15T22:43:00Z BEAM-2253 Use resource transformer plugin to use versions defined in root pom.xml within maven archetype poms
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/beam/pull/3181

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/beam/pull/3181
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3292

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3292
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3388

          BEAM-79 Sync gearpump-runner branch with master

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam sync-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3388.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3388


          commit 8362bdb9cd35cc02ed179b3a64fd72f1264a99be
          Author: JingsongLi <lzljs3620320@aliyun.com>
          Date: 2017-06-07T17:31:34Z

          BEAM-2423 Abstract StateInternalsTest for the different state internals

          commit f9d51aa5cc718c44c82ec4773dbd16a57e7d394e
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-06-12T20:35:35Z

          This closes #3313

          commit 809f17876d847002ba76979cb3362451fa01c110
          Author: Charles Chen <ccy@google.com>
          Date: 2017-06-12T21:17:50Z

          Reverse removal of NativeWrite evaluator in Python DirectRunner

          commit 86e04893a14d9b6aa29b78e25dc064879b712fb3
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-12T22:43:44Z

          This closes #3346

          commit f0f98c70b0b01c43ce1b093c2035b20bd90ba907
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-06-12T18:07:47Z

          Cleanup Combine Tests with Context

          Split out the "shared" bit of all the accumulators, so they show up as
          an explicit component of the final result string. Update timestamped
          creation logic.

          commit ce3dd4583294fcbb65121f5fecfc49a8b85634ad
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-06-12T22:48:38Z

          This closes #3345

          commit 1ac18b2eb1371422e60d50a8c3f37b3b24d59611
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-06-12T23:55:59Z

          Check for Deferral on Non-additional inputs

          Because Side Inputs are represented within the expanded inputs, the
          check that the transform is a Combine with Side Inputs would never be
          hit. This ensures that we do not consider additional inputs during the
          check to defer evaluation of the node.

          commit 32d55323cd5a88f604ad9c763ab265de845df08c
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-13T03:20:40Z

          This closes #3347: Check for Deferral on Non-additional inputs

          commit 6859f80400bb16bbca34ed282e7a5e8a1328f955
          Author: Ismaël Mejía <iemejia@apache.org>
          Date: 2017-06-05T21:20:27Z

          BEAM-2412 Update HBaseIO to use HBase client 1.2.6

          commit fe3d55403334bf235952fff33e6a90c0f4445a0d
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-06-13T05:59:30Z

          BEAM-2412 This closes #3327

          commit 4c36508733a69fafce0f7dfb86c71eee5eb6bc84
          Author: JingsongLi <lzljs3620320@aliyun.com>
          Date: 2017-06-07T06:34:25Z

          Use CoderTypeSerializer and remove unuse code in FlinkStateInternals

          commit 10b166b355a03daeae78dd1e71016fc72805939d
          Author: JingsongLi <lzljs3620320@aliyun.com>
          Date: 2017-06-07T06:40:30Z

          BEAM-1483 Support SetState in Flink runner and fix MapState to be consistent with InMemoryStateInternals.

          commit 4d18606378f43c7b0d3ac05d45ca6e0570e49eef
          Author: JingsongLi <lzljs3620320@aliyun.com>
          Date: 2017-06-13T02:15:33Z

          Add set and map readable test to StateInternalsTest

          commit 7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-06-13T09:35:43Z

          This closes #3309

          commit e71eb66ae319bdf0cdad1fe9b54662962c8e8f16
          Author: Robert Bradshaw <robertwb@google.com>
          Date: 2017-06-09T23:44:55Z

          Actually test the fn_api_runner.

          The test suite was not being run due to a typo.
          Fix breakage due to changes in the code in the meantime.

          commit bd83612f246c1654b80df19890a25f8de48a063a
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-06-13T15:56:36Z

          Closes #3339

          commit b547b5a1eee61e293ff2f8ccfac57f308867328c
          Author: Maria Garcia Herrero <mariagh@google.com>
          Date: 2017-06-10T06:34:59Z

          Make unique test names for value-provider arguments

          commit e906fe9c3a1e12f01d84e05fbb872d47b28d2fe9
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-13T16:34:16Z

          This closes #3341

          commit ee728f1b2f617dac8e5cd729cacf1a46911021e0
          Author: Vikas Kedigehalli <vikasrk@google.com>
          Date: 2017-06-13T06:11:22Z

          Fix WindowValueCoder for large timestamps

          commit c33e9b446ef071de02db77883517b308a205c9ff
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-13T17:11:33Z

          This closes #3350

          commit 33662b92f1a9936f594223c9aee1a7233f59a569
          Author: chamikara@google.com <chamikara@google.com>
          Date: 2017-06-08T21:56:24Z

          Adds ability to dynamically replace PTransforms during runtime.

          To this end, adds two interfaces, PTransformMatcher and PTransformOverride.

          Currently only supports replacements where input and output types are an exact match (we have to address complexities due to type hints before supporting replacements with different types).

          This will be used by SplittableDoFn where matching ParDo transforms will be dynamically replaced by SplittableParDo.

          commit 7d0f24a0d91fb1f45027f794d6ba00082f2332ce
          Author: chamikara@google.com <chamikara@google.com>
          Date: 2017-06-13T18:47:28Z

          This closes #3333

          commit 3a48e47c73520580021c21a037ab412a761d0eeb
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-06-12T18:51:39Z

          Improves message when transitively serializing PipelineOptions

          commit 996e35c1de8310a265c890cf8ca8e92416b3d16b
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-06-13T20:04:28Z

          This closes #3344: Improves message when transitively serializing PipelineOptions

          commit 9115af488ceb907de121313ffa096d58a0ccc1e1
          Author: Mairbek Khadikov <mairbek@google.com>
          Date: 2017-06-07T23:27:01Z

          SpannerIO: Introduced a MutationGroup.

          Allows to group together mutation in a logical bundle that is submitted in the same transaction.

          commit 646caf255510c735488b88ecc69edd9b9bde4081
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-06-13T20:28:47Z

          This closes #3319: BEAM-1542 SpannerIO: Introduced a MutationGroup

          commit 7e4e51fda633d208c4cc5e88182e5db16156f2cb
          Author: Charles Chen <ccy@google.com>
          Date: 2017-06-13T22:03:15Z

          Move Runner API protos to portability/api

          commit eebff9089e441599b70528b4897c2db1535626c3
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-06-13T23:25:38Z

          Closes #3354

          commit 8a850af3304a48618739dc23e286800dc0c4641a
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-06-13T19:50:58Z

          Do not produce Unprocessed Inputs if all inputs were Processed

          This stops the WatermarkManager "Pending Bundles" from growing without
          bound.

          commit dd9abc397c67b47f95414bfd5debb08b7b6a6eaf
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-06-14T17:21:02Z

          This closes #3353


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3388 BEAM-79 Sync gearpump-runner branch with master Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam sync-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3388.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3388 commit 8362bdb9cd35cc02ed179b3a64fd72f1264a99be Author: JingsongLi <lzljs3620320@aliyun.com> Date: 2017-06-07T17:31:34Z BEAM-2423 Abstract StateInternalsTest for the different state internals commit f9d51aa5cc718c44c82ec4773dbd16a57e7d394e Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-06-12T20:35:35Z This closes #3313 commit 809f17876d847002ba76979cb3362451fa01c110 Author: Charles Chen <ccy@google.com> Date: 2017-06-12T21:17:50Z Reverse removal of NativeWrite evaluator in Python DirectRunner commit 86e04893a14d9b6aa29b78e25dc064879b712fb3 Author: Ahmet Altay <altay@google.com> Date: 2017-06-12T22:43:44Z This closes #3346 commit f0f98c70b0b01c43ce1b093c2035b20bd90ba907 Author: Thomas Groh <tgroh@google.com> Date: 2017-06-12T18:07:47Z Cleanup Combine Tests with Context Split out the "shared" bit of all the accumulators, so they show up as an explicit component of the final result string. Update timestamped creation logic. commit ce3dd4583294fcbb65121f5fecfc49a8b85634ad Author: Thomas Groh <tgroh@google.com> Date: 2017-06-12T22:48:38Z This closes #3345 commit 1ac18b2eb1371422e60d50a8c3f37b3b24d59611 Author: Thomas Groh <tgroh@google.com> Date: 2017-06-12T23:55:59Z Check for Deferral on Non-additional inputs Because Side Inputs are represented within the expanded inputs, the check that the transform is a Combine with Side Inputs would never be hit. This ensures that we do not consider additional inputs during the check to defer evaluation of the node. commit 32d55323cd5a88f604ad9c763ab265de845df08c Author: Kenneth Knowles <klk@google.com> Date: 2017-06-13T03:20:40Z This closes #3347: Check for Deferral on Non-additional inputs commit 6859f80400bb16bbca34ed282e7a5e8a1328f955 Author: Ismaël Mejía <iemejia@apache.org> Date: 2017-06-05T21:20:27Z BEAM-2412 Update HBaseIO to use HBase client 1.2.6 commit fe3d55403334bf235952fff33e6a90c0f4445a0d Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-06-13T05:59:30Z BEAM-2412 This closes #3327 commit 4c36508733a69fafce0f7dfb86c71eee5eb6bc84 Author: JingsongLi <lzljs3620320@aliyun.com> Date: 2017-06-07T06:34:25Z Use CoderTypeSerializer and remove unuse code in FlinkStateInternals commit 10b166b355a03daeae78dd1e71016fc72805939d Author: JingsongLi <lzljs3620320@aliyun.com> Date: 2017-06-07T06:40:30Z BEAM-1483 Support SetState in Flink runner and fix MapState to be consistent with InMemoryStateInternals. commit 4d18606378f43c7b0d3ac05d45ca6e0570e49eef Author: JingsongLi <lzljs3620320@aliyun.com> Date: 2017-06-13T02:15:33Z Add set and map readable test to StateInternalsTest commit 7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-06-13T09:35:43Z This closes #3309 commit e71eb66ae319bdf0cdad1fe9b54662962c8e8f16 Author: Robert Bradshaw <robertwb@google.com> Date: 2017-06-09T23:44:55Z Actually test the fn_api_runner. The test suite was not being run due to a typo. Fix breakage due to changes in the code in the meantime. commit bd83612f246c1654b80df19890a25f8de48a063a Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-06-13T15:56:36Z Closes #3339 commit b547b5a1eee61e293ff2f8ccfac57f308867328c Author: Maria Garcia Herrero <mariagh@google.com> Date: 2017-06-10T06:34:59Z Make unique test names for value-provider arguments commit e906fe9c3a1e12f01d84e05fbb872d47b28d2fe9 Author: Ahmet Altay <altay@google.com> Date: 2017-06-13T16:34:16Z This closes #3341 commit ee728f1b2f617dac8e5cd729cacf1a46911021e0 Author: Vikas Kedigehalli <vikasrk@google.com> Date: 2017-06-13T06:11:22Z Fix WindowValueCoder for large timestamps commit c33e9b446ef071de02db77883517b308a205c9ff Author: Ahmet Altay <altay@google.com> Date: 2017-06-13T17:11:33Z This closes #3350 commit 33662b92f1a9936f594223c9aee1a7233f59a569 Author: chamikara@google.com <chamikara@google.com> Date: 2017-06-08T21:56:24Z Adds ability to dynamically replace PTransforms during runtime. To this end, adds two interfaces, PTransformMatcher and PTransformOverride. Currently only supports replacements where input and output types are an exact match (we have to address complexities due to type hints before supporting replacements with different types). This will be used by SplittableDoFn where matching ParDo transforms will be dynamically replaced by SplittableParDo. commit 7d0f24a0d91fb1f45027f794d6ba00082f2332ce Author: chamikara@google.com <chamikara@google.com> Date: 2017-06-13T18:47:28Z This closes #3333 commit 3a48e47c73520580021c21a037ab412a761d0eeb Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-06-12T18:51:39Z Improves message when transitively serializing PipelineOptions commit 996e35c1de8310a265c890cf8ca8e92416b3d16b Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-06-13T20:04:28Z This closes #3344: Improves message when transitively serializing PipelineOptions commit 9115af488ceb907de121313ffa096d58a0ccc1e1 Author: Mairbek Khadikov <mairbek@google.com> Date: 2017-06-07T23:27:01Z SpannerIO: Introduced a MutationGroup. Allows to group together mutation in a logical bundle that is submitted in the same transaction. commit 646caf255510c735488b88ecc69edd9b9bde4081 Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-06-13T20:28:47Z This closes #3319: BEAM-1542 SpannerIO: Introduced a MutationGroup commit 7e4e51fda633d208c4cc5e88182e5db16156f2cb Author: Charles Chen <ccy@google.com> Date: 2017-06-13T22:03:15Z Move Runner API protos to portability/api commit eebff9089e441599b70528b4897c2db1535626c3 Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-06-13T23:25:38Z Closes #3354 commit 8a850af3304a48618739dc23e286800dc0c4641a Author: Thomas Groh <tgroh@google.com> Date: 2017-06-13T19:50:58Z Do not produce Unprocessed Inputs if all inputs were Processed This stops the WatermarkManager "Pending Bundles" from growing without bound. commit dd9abc397c67b47f95414bfd5debb08b7b6a6eaf Author: Thomas Groh <tgroh@google.com> Date: 2017-06-14T17:21:02Z This closes #3353
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3388

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3388
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3479

          BEAM-79 Merge master into gearpump-runner branch

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam sync-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3479.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3479


          commit cab4d8969e7f95b0ece59838ad2d578e75d38823
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-22T03:25:31Z

          DataflowRunner: Reject merging windowing for stateful ParDo

          commit 5f8cfa7412e8f4ce97d6df4413fecbea22b4b5e1
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-23T03:05:28Z

          This closes #3419: DataflowRunner: Reject merging windowing for stateful ParDo

          commit 649994b353afe28c917969609c7a1a47a4f39aaf
          Author: Rune Fevang <fevang@exabel.com>
          Date: 2017-06-15T11:51:12Z

          Allow output from FinishBundle in DoFnTester

          commit b3c36256e32f4a4a616d0e152686d29f1174c5cd
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-23T03:07:48Z

          This closes #3367: Allow output from FinishBundle in DoFnTester

          commit 8dcda6e40355af13f4d92fcd44aae4539a225a4a
          Author: Sourabh Bajaj <sourabhbajaj@google.com>
          Date: 2017-06-23T00:08:20Z

          BEAM-2497 Fix the reading of concat gzip files

          commit 336b7f1cffc8709cd480a243a16de95bce39b53e
          Author: Chamikara Jayalath <chamikara@google.com>
          Date: 2017-06-23T05:13:05Z

          This closes #3428

          commit f291713b28e3ba0246a8c0a710c71506cd0a0f91
          Author: Etienne Chauchot <echauchot@gmail.com>
          Date: 2017-06-21T08:39:39Z

          BEAM-2489 Use dynamic ES port in HIFIOWithElasticTest

          commit 6cef5c7b625c6b5c2078084ba611b1c2adccc998
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-06-23T13:01:42Z

          BEAM-2489 This closes #3411

          commit fd8f15f1ac761425dc791a455b042a8846081f48
          Author: Mark Liu <markliu@google.com>
          Date: 2017-06-22T21:04:00Z

          BEAM-2745 Add Jenkins Suite for Python Performance Test

          commit af69e979ab3037274bdd78904bdeb5016563bcad
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-23T21:03:17Z

          This closes #3426

          commit 32095487e56b63b5c1aa690bb6e098375cb108d5
          Author: Vikas Kedigehalli <vikasrk@google.com>
          Date: 2017-06-23T18:50:12Z

          Fix python fn API data plane remote grpc port access

          commit 9acce7150ee8fa6c9e50049155ad7b85b646f98e
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-23T22:13:30Z

          This closes #3431

          commit 903da41ac5395e76c44ef8ae1c8a695569e23abb
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-06-23T22:01:42Z

          Avoid pickling the entire pipeline per-transform.

          commit a90e40ae9f748c1c8392c1198469f3229a06ed70
          Author: Robert Bradshaw <robertwb@gmail.com>
          Date: 2017-06-23T23:39:52Z

          Closes #3433

          commit e45f522d6e945899c20259ebf8faca105c2e552e
          Author: Valentyn Tymofieiev <valentyn@google.com>
          Date: 2017-06-23T23:44:49Z

          Fix a typo in function args

          commit 16f87f49f20796e29d01ed363a9097ea5420583c
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-24T00:34:21Z

          This closes #3435

          commit 926f949580c3a21df72a8836feda1f6b947850ec
          Author: Charles Chen <ccy@google.com>
          Date: 2017-06-26T20:00:14Z

          Remove old deprecated PubSub code

          commit 1ea1de4aa9d32e3c5a596ccd7d84aff1cc2a7428
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-26T21:23:08Z

          This closes #3440

          commit bec32fe93c6b5c16563d7ea4b877a2dee3352fee
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-06-16T21:56:07Z

          Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part)

          commit 2052cc7689b4ed53f817f56dd71a32235fb083ca
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-06-23T17:16:30Z

          Bump Dataflow worker to 0623

          commit 95e6bbe5065a983f9c71b79b20df1fda83c9fd1b
          Author: Eugene Kirpichov <kirpichov@google.com>
          Date: 2017-06-27T00:25:14Z

          This closes #3383: BEAM-2447 Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part)

          commit eb379e76adaa9c4b4e24a4b3c5757be8523d95c4
          Author: Charles Chen <ccy@google.com>
          Date: 2017-06-26T23:54:00Z

          Implement streaming GroupByKey in Python DirectRunner

          commit 8036001da6f90eac20d787a05d35a51e30146278
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-06-27T05:49:19Z

          This closes #3444

          commit cbfcad823972270171552e556b7fa8d8d4882f14
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2016-12-28T14:06:09Z

          BEAM-1237 Create AmqpIO

          commit 9df865add9bdd83d265d0f8442e6b07467bcf17d
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-06-27T13:33:51Z

          BEAM-1237 This closes #1725

          commit 01b3febc62debf8b4fd1582225d59768d231601a
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-06-26T12:55:13Z

          BEAM-2514 Improve error message on missing required value

          commit 99ef92bc97692bdba3ea8e10cc09d81f3d64b69b
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-06-27T16:23:03Z

          BEAM-2514 Improve error message on missing required value

          This closes #3438

          commit d33d64644c8ddc74a3dc07bf0c642a9ed818a624
          Author: Nigel Kilmer <nkilmer@google.com>
          Date: 2017-06-21T18:26:10Z

          Removed uses of proto builder clone method

          commit e93c06485508ae3a7a8304be3d6127d604d9356e
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-06-27T16:40:53Z

          BEAM-2392 Remove uses of proto builder clone

          This closes #3415

          commit 8dd0077d2a58e278b11c7e7eb4b5f182e1400992
          Author: Vikas Kedigehalli <vikasrk@google.com>
          Date: 2017-06-27T01:47:39Z

          Enable grpc controller in fn_api_runner


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3479 BEAM-79 Merge master into gearpump-runner branch Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam sync-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3479.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3479 commit cab4d8969e7f95b0ece59838ad2d578e75d38823 Author: Kenneth Knowles <klk@google.com> Date: 2017-06-22T03:25:31Z DataflowRunner: Reject merging windowing for stateful ParDo commit 5f8cfa7412e8f4ce97d6df4413fecbea22b4b5e1 Author: Kenneth Knowles <klk@google.com> Date: 2017-06-23T03:05:28Z This closes #3419: DataflowRunner: Reject merging windowing for stateful ParDo commit 649994b353afe28c917969609c7a1a47a4f39aaf Author: Rune Fevang <fevang@exabel.com> Date: 2017-06-15T11:51:12Z Allow output from FinishBundle in DoFnTester commit b3c36256e32f4a4a616d0e152686d29f1174c5cd Author: Kenneth Knowles <klk@google.com> Date: 2017-06-23T03:07:48Z This closes #3367: Allow output from FinishBundle in DoFnTester commit 8dcda6e40355af13f4d92fcd44aae4539a225a4a Author: Sourabh Bajaj <sourabhbajaj@google.com> Date: 2017-06-23T00:08:20Z BEAM-2497 Fix the reading of concat gzip files commit 336b7f1cffc8709cd480a243a16de95bce39b53e Author: Chamikara Jayalath <chamikara@google.com> Date: 2017-06-23T05:13:05Z This closes #3428 commit f291713b28e3ba0246a8c0a710c71506cd0a0f91 Author: Etienne Chauchot <echauchot@gmail.com> Date: 2017-06-21T08:39:39Z BEAM-2489 Use dynamic ES port in HIFIOWithElasticTest commit 6cef5c7b625c6b5c2078084ba611b1c2adccc998 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-06-23T13:01:42Z BEAM-2489 This closes #3411 commit fd8f15f1ac761425dc791a455b042a8846081f48 Author: Mark Liu <markliu@google.com> Date: 2017-06-22T21:04:00Z BEAM-2745 Add Jenkins Suite for Python Performance Test commit af69e979ab3037274bdd78904bdeb5016563bcad Author: Ahmet Altay <altay@google.com> Date: 2017-06-23T21:03:17Z This closes #3426 commit 32095487e56b63b5c1aa690bb6e098375cb108d5 Author: Vikas Kedigehalli <vikasrk@google.com> Date: 2017-06-23T18:50:12Z Fix python fn API data plane remote grpc port access commit 9acce7150ee8fa6c9e50049155ad7b85b646f98e Author: Ahmet Altay <altay@google.com> Date: 2017-06-23T22:13:30Z This closes #3431 commit 903da41ac5395e76c44ef8ae1c8a695569e23abb Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-06-23T22:01:42Z Avoid pickling the entire pipeline per-transform. commit a90e40ae9f748c1c8392c1198469f3229a06ed70 Author: Robert Bradshaw <robertwb@gmail.com> Date: 2017-06-23T23:39:52Z Closes #3433 commit e45f522d6e945899c20259ebf8faca105c2e552e Author: Valentyn Tymofieiev <valentyn@google.com> Date: 2017-06-23T23:44:49Z Fix a typo in function args commit 16f87f49f20796e29d01ed363a9097ea5420583c Author: Ahmet Altay <altay@google.com> Date: 2017-06-24T00:34:21Z This closes #3435 commit 926f949580c3a21df72a8836feda1f6b947850ec Author: Charles Chen <ccy@google.com> Date: 2017-06-26T20:00:14Z Remove old deprecated PubSub code commit 1ea1de4aa9d32e3c5a596ccd7d84aff1cc2a7428 Author: Ahmet Altay <altay@google.com> Date: 2017-06-26T21:23:08Z This closes #3440 commit bec32fe93c6b5c16563d7ea4b877a2dee3352fee Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-06-16T21:56:07Z Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part) commit 2052cc7689b4ed53f817f56dd71a32235fb083ca Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-06-23T17:16:30Z Bump Dataflow worker to 0623 commit 95e6bbe5065a983f9c71b79b20df1fda83c9fd1b Author: Eugene Kirpichov <kirpichov@google.com> Date: 2017-06-27T00:25:14Z This closes #3383: BEAM-2447 Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part) commit eb379e76adaa9c4b4e24a4b3c5757be8523d95c4 Author: Charles Chen <ccy@google.com> Date: 2017-06-26T23:54:00Z Implement streaming GroupByKey in Python DirectRunner commit 8036001da6f90eac20d787a05d35a51e30146278 Author: Ahmet Altay <altay@google.com> Date: 2017-06-27T05:49:19Z This closes #3444 commit cbfcad823972270171552e556b7fa8d8d4882f14 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2016-12-28T14:06:09Z BEAM-1237 Create AmqpIO commit 9df865add9bdd83d265d0f8442e6b07467bcf17d Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-06-27T13:33:51Z BEAM-1237 This closes #1725 commit 01b3febc62debf8b4fd1582225d59768d231601a Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-06-26T12:55:13Z BEAM-2514 Improve error message on missing required value commit 99ef92bc97692bdba3ea8e10cc09d81f3d64b69b Author: Luke Cwik <lcwik@google.com> Date: 2017-06-27T16:23:03Z BEAM-2514 Improve error message on missing required value This closes #3438 commit d33d64644c8ddc74a3dc07bf0c642a9ed818a624 Author: Nigel Kilmer <nkilmer@google.com> Date: 2017-06-21T18:26:10Z Removed uses of proto builder clone method commit e93c06485508ae3a7a8304be3d6127d604d9356e Author: Luke Cwik <lcwik@google.com> Date: 2017-06-27T16:40:53Z BEAM-2392 Remove uses of proto builder clone This closes #3415 commit 8dd0077d2a58e278b11c7e7eb4b5f182e1400992 Author: Vikas Kedigehalli <vikasrk@google.com> Date: 2017-06-27T01:47:39Z Enable grpc controller in fn_api_runner
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3479

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3479
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3515

          BEAM-79 Sync gearpump-runner with master and upgrade to gearpump 0.8.4

          Be sure to do all of the following to help us incorporate your contribution
          quickly and easily:

          • [ ] Make sure the PR title is formatted like:
            `BEAM-<Jira issue #> Description of pull request`
          • [ ] Make sure tests pass via `mvn clean verify`.
          • [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
            number, if there is one.
          • [ ] If this contribution is large, please file an Apache
            [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam sync-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3515.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3515


          commit 7b8cd6401cb5ed6e184ed36571a89d3ae324dd5f
          Author: Jeremie Lenfant-Engelmann <jeremiele@google.com>
          Date: 2017-06-29T01:32:56Z

          Properly convert milliseconds whether there's less than 3/more than 9 digits.
          TimeUtil did not properly convert (and returned null) when the number of
          digits for fractions of seconds was less than 3 digits or more than 9 digits.
          The solution is to pad with zeros when there is less than 3 digits and to
          truncate when there is more than 3.

          commit f75dfe711b357800762a93dc5059d9a19da9ee61
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-06-30T16:00:13Z

          Properly convert milliseconds whether there's less than 3/more than 9 digits.

          This closes #3469

          commit 56cb6c51748fde6ad56522733ab10edca062e802
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-13T17:29:50Z

          Add support for PipelineOptions parameters

          This is a step towards eliminating catch-all context parameters and
          making DoFns express their fine-grained data needs.

          commit 0e429b33ff85eba08da5018c9febd0b99b44f720
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-30T22:39:20Z

          This closes #3355: Add support for PipelineOptions parameters

          commit f99ab1a472868e4ce175a86e5d76823b1c09c10b
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-07-01T04:42:17Z

          Fix DoFn javadoc: StateSpec does not require a key

          commit ce4e51747501111ae2c4b1691c6994bd0f92e161
          Author: Ismaël Mejía <iemejia@apache.org>
          Date: 2017-06-04T20:55:05Z

          Make modules that depend on Hadoop and Spark use the same version property

          commit 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-07-03T07:26:40Z

          This closes #3291

          commit 75475ef3dc23a09fa9bbba478d6fdbc468f7dd2e
          Author: Ismaël Mejía <iemejia@gmail.com>
          Date: 2017-06-28T14:58:55Z

          BEAM-2530 Fix compilation of modules with Java 9 that depend on jdk.tools

          commit 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-07-03T12:23:18Z

          BEAM-2530 This closes #3460

          commit 68f1fb64fd2565e287e322d715ca778d01e7137b
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-07-01T00:37:33Z

          For GCS operations use an http client with a default timeout value.

          commit b8ac32641183b06a87517a169c8ecc5371996aed
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-07-04T18:07:31Z

          This closes #3486

          commit 51877a3405dbf778c3bb88f19bb194e54c3b3def
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-07-05T14:47:29Z

          [maven-release-plugin] prepare branch release-2.1.0

          commit 7f0723cf7fd587c4f8dbe8a8b1c9d298a7b1e5e3
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-07-05T14:47:38Z

          [maven-release-plugin] prepare for next development iteration

          commit 14fa7f79f0830739122b7573e032ad0aea172a98
          Author: Jean-Baptiste Onofré <jbonofre@apache.org>
          Date: 2017-07-05T14:52:48Z

          Update Python SDK version

          commit 06897b1cc142f658437ac7779c849e5182e331f1
          Author: Jason Kuster <jasonkuster@google.com>
          Date: 2017-06-09T08:39:15Z

          Website Mergebot Job

          Signed-off-by: Jason Kuster <jasonkuster@google.com>

          commit f0549b464737d0d11ae1da8e0534622b10247208
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-07-05T16:22:20Z

          This closes #3485

          commit 6ca410a908f1f4e7ac1e141ee1335f7a537bb150
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-07-05T17:38:44Z

          BEAM-2553 Update Maven exec plugin to 1.6.0 to incorporate messaging improvements

          commit 39c80f640749fed5f55a0c8f25c849e5f7c3d179
          Author: Luke Cwik <lcwik@google.com>
          Date: 2017-07-05T19:26:28Z

          BEAM-2553 Update Maven exec plugin to 1.6.0 to incorporate messaging improvements

          This closes #3494

          commit 29c2bca4649317f2ebb1c89f92bf97fbb27602ca
          Author: Thomas Groh <tgroh@google.com>
          Date: 2017-07-05T21:16:50Z

          Disallow Combiner Lifting for multi-window WindowFns

          commit 63f66d6d81652dba5a7804648cde0e7debffd0f6
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-07-05T22:00:09Z

          This closes #3489: Fix DoFn javadoc: StateSpec does not require a key

          commit 23e385faa193a00f9b10e3f8f0afe832087bff06
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-07-05T21:34:07Z

          Update SDK dependencies

          commit ea9f7fbdde5ede5e5f5060da5dfe2749992a2fa7
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-07-05T23:11:46Z

          This closes #3497

          commit e8fdd373c7b86652fda94ff76ef7246357e6288c
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-07-05T23:41:16Z

          This closes #3496: Disallow Combiner Lifting for multi-window WindowFns

          commit a75202f344f22be5c5fdf62b3eb54a151ad29af6
          Author: Charles Chen <ccy@google.com>
          Date: 2017-07-05T23:18:51Z

          Fix PValue input in _PubSubReadEvaluator

          commit bf6dda32047e0bfba18de5aa27b1b6a593f86fc9
          Author: Ahmet Altay <altay@google.com>
          Date: 2017-07-06T01:09:05Z

          This closes #3499

          commit 2259c309c5b81a5d1e32732dd35e1102766401fa
          Author: Raghu Angadi <rangadi@google.com>
          Date: 2017-06-28T19:07:06Z

          BEAM-2534 Handle offset gaps in Kafka messages.

          KafkaIO logged a warning when there is a gap in offstes for messages.
          Kafka also support 'KV' store style topics where some of the messages
          are deleted leading gaps in offsets. This PR removes the log and
          accounts for offset gaps in backlog estimate.

          commit 85a99e29448670ae6728a8ee2e4cd3ef95877c3e
          Author: JingsongLi <lzljs3620320@aliyun.com>
          Date: 2017-07-06T16:48:36Z

          This closes #3461

          commit 526037b6786315b9f9fdca6edb636baeb6f83e3f
          Author: Raghu Angadi <rangadi@google.com>
          Date: 2017-07-04T06:54:10Z

          Add timeout to initialization of partition in KafkaIO

          commit b8f8d18ae2cdbb4874d9a0f45038037ecc2381d1
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-07-06T18:59:18Z

          This closes #3492: BEAM-2551 KafkaIO reader blocks indefinitely in case of network issues

          Add timeout to initialization of partition in KafkaIO

          commit bd631b89a8434f0756e1596875e89013fb623ab5
          Author: Kenneth Knowles <klk@google.com>
          Date: 2017-06-23T01:09:11Z

          Ignore processing time timers in expired windows


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3515 BEAM-79 Sync gearpump-runner with master and upgrade to gearpump 0.8.4 Be sure to do all of the following to help us incorporate your contribution quickly and easily: [ ] Make sure the PR title is formatted like: ` BEAM-<Jira issue #> Description of pull request` [ ] Make sure tests pass via `mvn clean verify`. [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam sync-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3515.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3515 commit 7b8cd6401cb5ed6e184ed36571a89d3ae324dd5f Author: Jeremie Lenfant-Engelmann <jeremiele@google.com> Date: 2017-06-29T01:32:56Z Properly convert milliseconds whether there's less than 3/more than 9 digits. TimeUtil did not properly convert (and returned null) when the number of digits for fractions of seconds was less than 3 digits or more than 9 digits. The solution is to pad with zeros when there is less than 3 digits and to truncate when there is more than 3. commit f75dfe711b357800762a93dc5059d9a19da9ee61 Author: Luke Cwik <lcwik@google.com> Date: 2017-06-30T16:00:13Z Properly convert milliseconds whether there's less than 3/more than 9 digits. This closes #3469 commit 56cb6c51748fde6ad56522733ab10edca062e802 Author: Kenneth Knowles <klk@google.com> Date: 2017-06-13T17:29:50Z Add support for PipelineOptions parameters This is a step towards eliminating catch-all context parameters and making DoFns express their fine-grained data needs. commit 0e429b33ff85eba08da5018c9febd0b99b44f720 Author: Kenneth Knowles <klk@google.com> Date: 2017-06-30T22:39:20Z This closes #3355: Add support for PipelineOptions parameters commit f99ab1a472868e4ce175a86e5d76823b1c09c10b Author: Kenneth Knowles <klk@google.com> Date: 2017-07-01T04:42:17Z Fix DoFn javadoc: StateSpec does not require a key commit ce4e51747501111ae2c4b1691c6994bd0f92e161 Author: Ismaël Mejía <iemejia@apache.org> Date: 2017-06-04T20:55:05Z Make modules that depend on Hadoop and Spark use the same version property commit 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-07-03T07:26:40Z This closes #3291 commit 75475ef3dc23a09fa9bbba478d6fdbc468f7dd2e Author: Ismaël Mejía <iemejia@gmail.com> Date: 2017-06-28T14:58:55Z BEAM-2530 Fix compilation of modules with Java 9 that depend on jdk.tools commit 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-07-03T12:23:18Z BEAM-2530 This closes #3460 commit 68f1fb64fd2565e287e322d715ca778d01e7137b Author: Ahmet Altay <altay@google.com> Date: 2017-07-01T00:37:33Z For GCS operations use an http client with a default timeout value. commit b8ac32641183b06a87517a169c8ecc5371996aed Author: Ahmet Altay <altay@google.com> Date: 2017-07-04T18:07:31Z This closes #3486 commit 51877a3405dbf778c3bb88f19bb194e54c3b3def Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-07-05T14:47:29Z [maven-release-plugin] prepare branch release-2.1.0 commit 7f0723cf7fd587c4f8dbe8a8b1c9d298a7b1e5e3 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-07-05T14:47:38Z [maven-release-plugin] prepare for next development iteration commit 14fa7f79f0830739122b7573e032ad0aea172a98 Author: Jean-Baptiste Onofré <jbonofre@apache.org> Date: 2017-07-05T14:52:48Z Update Python SDK version commit 06897b1cc142f658437ac7779c849e5182e331f1 Author: Jason Kuster <jasonkuster@google.com> Date: 2017-06-09T08:39:15Z Website Mergebot Job Signed-off-by: Jason Kuster <jasonkuster@google.com> commit f0549b464737d0d11ae1da8e0534622b10247208 Author: Thomas Groh <tgroh@google.com> Date: 2017-07-05T16:22:20Z This closes #3485 commit 6ca410a908f1f4e7ac1e141ee1335f7a537bb150 Author: Luke Cwik <lcwik@google.com> Date: 2017-07-05T17:38:44Z BEAM-2553 Update Maven exec plugin to 1.6.0 to incorporate messaging improvements commit 39c80f640749fed5f55a0c8f25c849e5f7c3d179 Author: Luke Cwik <lcwik@google.com> Date: 2017-07-05T19:26:28Z BEAM-2553 Update Maven exec plugin to 1.6.0 to incorporate messaging improvements This closes #3494 commit 29c2bca4649317f2ebb1c89f92bf97fbb27602ca Author: Thomas Groh <tgroh@google.com> Date: 2017-07-05T21:16:50Z Disallow Combiner Lifting for multi-window WindowFns commit 63f66d6d81652dba5a7804648cde0e7debffd0f6 Author: Kenneth Knowles <klk@google.com> Date: 2017-07-05T22:00:09Z This closes #3489: Fix DoFn javadoc: StateSpec does not require a key commit 23e385faa193a00f9b10e3f8f0afe832087bff06 Author: Ahmet Altay <altay@google.com> Date: 2017-07-05T21:34:07Z Update SDK dependencies commit ea9f7fbdde5ede5e5f5060da5dfe2749992a2fa7 Author: Ahmet Altay <altay@google.com> Date: 2017-07-05T23:11:46Z This closes #3497 commit e8fdd373c7b86652fda94ff76ef7246357e6288c Author: Kenneth Knowles <klk@google.com> Date: 2017-07-05T23:41:16Z This closes #3496: Disallow Combiner Lifting for multi-window WindowFns commit a75202f344f22be5c5fdf62b3eb54a151ad29af6 Author: Charles Chen <ccy@google.com> Date: 2017-07-05T23:18:51Z Fix PValue input in _PubSubReadEvaluator commit bf6dda32047e0bfba18de5aa27b1b6a593f86fc9 Author: Ahmet Altay <altay@google.com> Date: 2017-07-06T01:09:05Z This closes #3499 commit 2259c309c5b81a5d1e32732dd35e1102766401fa Author: Raghu Angadi <rangadi@google.com> Date: 2017-06-28T19:07:06Z BEAM-2534 Handle offset gaps in Kafka messages. KafkaIO logged a warning when there is a gap in offstes for messages. Kafka also support 'KV' store style topics where some of the messages are deleted leading gaps in offsets. This PR removes the log and accounts for offset gaps in backlog estimate. commit 85a99e29448670ae6728a8ee2e4cd3ef95877c3e Author: JingsongLi <lzljs3620320@aliyun.com> Date: 2017-07-06T16:48:36Z This closes #3461 commit 526037b6786315b9f9fdca6edb636baeb6f83e3f Author: Raghu Angadi <rangadi@google.com> Date: 2017-07-04T06:54:10Z Add timeout to initialization of partition in KafkaIO commit b8f8d18ae2cdbb4874d9a0f45038037ecc2381d1 Author: Kenneth Knowles <klk@google.com> Date: 2017-07-06T18:59:18Z This closes #3492: BEAM-2551 KafkaIO reader blocks indefinitely in case of network issues Add timeout to initialization of partition in KafkaIO commit bd631b89a8434f0756e1596875e89013fb623ab5 Author: Kenneth Knowles <klk@google.com> Date: 2017-06-23T01:09:11Z Ignore processing time timers in expired windows
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3515

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3515
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3611

          BEAM-79 merge gearpump-runner into master

          Follow this checklist to help us incorporate your contribution quickly and easily:

          • [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
          • [ ] Each commit in the pull request should have a meaningful subject line and body.
          • [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
          • [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
          • [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
          • [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam gearpump-runner

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3611.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3611


          commit 9478f4117de3a2d0ea40614ed4cb801918610724
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-03-15T08:15:16Z

          BEAM-79 add Gearpump runner

          commit 02b2248a5b3c8a2c064547d7380bebc97f849bf1
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-20T16:07:48Z

          This closes #323

          commit 2a0ba61e8507e1539115b583749a78f14d577bd8
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-08-25T18:36:45Z

          Merge branch master into gearpump-runner

          commit 1672b5483e029292816397248dc6fe63bf51f4af
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-07-23T06:10:15Z

          move integration tests to profile

          commit 276a2e106aa1a5736666fc2eb2426b640f63cf68
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-07-28T08:30:13Z

          add package-info.java

          commit 40be715a696bb1218b209f7ad9a979b7e5d088d3
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-08-10T17:26:57Z

          Update Gearpump runner version to 0.3.0-incubating

          commit bc1b354949416db3b52c4f37c66968bdb86f0813
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-08-11T23:22:00Z

          Rename DoFn to OldDoFn in Gearpump runner

          commit 091a15a07c7625ae3009cefaecece3a29a34c109
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-08-25T18:40:03Z

          This closess #750

          commit fb74c936ed92c7a8548c338cc03957794fc60902
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-08-26T23:25:58Z

          gearpump: switch to stable version

          They have apparently deleted the SNAPSHOT jar and now builds are failing.

          commit bf0a2edae11416a3cbddeaff2c0a70adc272c5fe
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-08-27T00:46:42Z

          Closes #895

          commit 89921c41ca9d4c333af45efa32359a631214c1df
          Author: bchambers <bchambers@google.com>
          Date: 2016-07-29T16:41:17Z

          Remove Counter and associated code

          Aggregator is the model level concept. Counter was specific to the
          Dataflow Runner, and is now not needed as part of Beam.

          commit 7fc2c6848f002ac8b2ccbe35e6b5a576777a7af9
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-03T00:25:14Z

          BEAM-495 Create General Verifier for File Checksum

          commit b47549e4893a6d487c00ea0ba02619168a3f19f3
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-03T00:47:46Z

          Add output checksum to WordCountITOptions

          commit 58cd781c82fa728f34f5ab0641f8f9b6edcf449c
          Author: Ian Zhou <ianzhou@google.com>
          Date: 2016-08-05T22:31:59Z

          Added unit tests and error handling in removeTemporaryTables

          commit 36a9aa232ea56de449930194788becce585212ef
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-09T02:09:58Z

          Improve Write Error Message

          If provided with an Unbounded PCollection, Write will fail due to
          restriction of calling finalize only once. This error message fails in a
          deep stack trace based on it not being possible to apply a GroupByKey.
          Instead, throw immediately on application with a specific error message.

          commit d5641553cebb02f08ca7c1fe667948d39cb3962c
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-09T17:47:09Z

          Remove Streaming Write Overrides in DataflowRunner

          These writes should be forbidden based on the boundedness of the input
          PCollection. As Write explicitly forbids the application of the
          transform to an Unbounded PCollection, this will be equivalent in most
          cases; In cases where the input PCollection is Bounded, due to an
          UnboundedReadFromBoundedSource, the write will function as expected and
          does not need to be forbidden.

          commit 011bea9a83a828e0d8c6518ab83aa5cc4f75e146
          Author: David Rieber <drieber@google.com>
          Date: 2016-08-09T21:05:25Z

          Do not add DataDisks to windmill service jobs.

          commit 0dfb8ff55d6f80264222fde4501ea3050d2e3911
          Author: gaurav gupta <gaugupt3@cisco.com>
          Date: 2016-08-10T23:43:03Z

          Made byteArrayCoder final static

          commit b9f826366823003940805e6469b10df8819b0977
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-08-11T00:58:09Z

          CompressedSource: CompressedReader is never splittable

          The only way it's safe to split a compressed file is if the file is not compressed. This can
          only happen when the source itself is splittable, and that in turn will result in the inner
          source's reader being returned. A CompressedReader will only be created in the event that
          the file is NOT splittable. So remove all the logic handling splittable compressed readers,
          and instead go with the logic when we know/assume the file is compressed.

          • TextIO: test compression with larger files

          It is important for correctness that we test with large files
          because otherwise the compressed file may be larger than the
          uncompressed file, which could mask bugs

          • TextIOTest: flesh out more
          • TextIOTest: add large uncompressed file

          commit 1d86335314685926d3eb0f9765d615a77cee75e6
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-11T16:16:55Z

          Remove timeout in DirectRunnerTest

          If the test hangs due to bugs, the infrastructure should kill it.

          commit 37ce2a3e75cb96a4b3fdcd4938fb7fda95122724
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-11T18:26:28Z

          More unit test and code style fix

          commit 046e36eaa41659ae43866bc6fbce4f122889f286
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-11T18:55:17Z

          Using IOChannelUtils to resolve file path

          commit d056f4661da2cc399cab44c6604eaa61d1dfd178
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-14T21:51:02Z

          Add DoFn @Setup and @Teardown

          Methods annotated with these annotations are used to perform expensive
          setup work and clean up a DoFn after another method throws an exception
          or the DoFn is discarded.

          commit b80d96748dcb71f93697126489c924020ebbd4a9
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-15T18:27:00Z

          Add TransformEvaluatorFactory#cleanup

          This cleans up any state stored within the Transform Evaluator Factory.

          commit 77c90d00ae715795e73efec8f8e85e3917cf8d80
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-19T18:03:15Z

          Replace CloningThreadLocal with DoFnLifecycleManager

          This is a more focused interface that interacts with a DoFn before it
          is available for use and after it has completed and the reference is
          lost. It is required to properly support setup and teardown, as the
          fields in a ThreadLocal cannot all be cleaned up without additional
          tracking.

          Part of BEAM-452.

          commit 39f763e16182e33019a1805d6210549934998856
          Author: Pei He <peihe@google.com>
          Date: 2016-08-01T20:41:59Z

          Remove DataflowPipelineJob from examples

          commit 6603307062ec99639d3e3e05aebc0d1ea32ad411
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-11T17:45:43Z

          Move ParDo Lifecycle tests to their own file

          These tests are not yet functional in all runners, and this makes them
          easier to ignore.

          commit d99a652f96f12bcc235caa038ffa741906336b1f
          Author: Maximilian Michels <mxm@apache.org>
          Date: 2016-08-12T15:51:02Z

          [flink] add missing maven config to example pom

          commit 424c4c492965e5a93b1c020c8d52805e3a9a9088
          Author: mariusz89016 <mariusz89016@gmail.com>
          Date: 2016-08-13T22:35:19Z

          BEAM-432 Corrected BigQueryIO javadoc

          commit d6cf4f2a3cda9ff4ff2da105c08be8101c58e6f1
          Author: Daniel Halperin <dhalperi@users.noreply.github.com>
          Date: 2016-08-15T22:21:15Z

          Exclude ParDoTest from Dataflow @RunnableOnService

          Until we implement it for Dataflow runner.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3611 BEAM-79 merge gearpump-runner into master Follow this checklist to help us incorporate your contribution quickly and easily: [ ] Make sure there is a [JIRA issue] ( https://issues.apache.org/jira/projects/BEAM/issues/ ) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. [ ] Each commit in the pull request should have a meaningful subject line and body. [ ] Format the pull request title like ` [BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam gearpump-runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3611.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3611 commit 9478f4117de3a2d0ea40614ed4cb801918610724 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-03-15T08:15:16Z BEAM-79 add Gearpump runner commit 02b2248a5b3c8a2c064547d7380bebc97f849bf1 Author: Kenneth Knowles <klk@google.com> Date: 2016-07-20T16:07:48Z This closes #323 commit 2a0ba61e8507e1539115b583749a78f14d577bd8 Author: Kenneth Knowles <klk@google.com> Date: 2016-08-25T18:36:45Z Merge branch master into gearpump-runner commit 1672b5483e029292816397248dc6fe63bf51f4af Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-07-23T06:10:15Z move integration tests to profile commit 276a2e106aa1a5736666fc2eb2426b640f63cf68 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-07-28T08:30:13Z add package-info.java commit 40be715a696bb1218b209f7ad9a979b7e5d088d3 Author: Kenneth Knowles <klk@google.com> Date: 2016-08-10T17:26:57Z Update Gearpump runner version to 0.3.0-incubating commit bc1b354949416db3b52c4f37c66968bdb86f0813 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-08-11T23:22:00Z Rename DoFn to OldDoFn in Gearpump runner commit 091a15a07c7625ae3009cefaecece3a29a34c109 Author: Kenneth Knowles <klk@google.com> Date: 2016-08-25T18:40:03Z This closess #750 commit fb74c936ed92c7a8548c338cc03957794fc60902 Author: Dan Halperin <dhalperi@google.com> Date: 2016-08-26T23:25:58Z gearpump: switch to stable version They have apparently deleted the SNAPSHOT jar and now builds are failing. commit bf0a2edae11416a3cbddeaff2c0a70adc272c5fe Author: Dan Halperin <dhalperi@google.com> Date: 2016-08-27T00:46:42Z Closes #895 commit 89921c41ca9d4c333af45efa32359a631214c1df Author: bchambers <bchambers@google.com> Date: 2016-07-29T16:41:17Z Remove Counter and associated code Aggregator is the model level concept. Counter was specific to the Dataflow Runner, and is now not needed as part of Beam. commit 7fc2c6848f002ac8b2ccbe35e6b5a576777a7af9 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-03T00:25:14Z BEAM-495 Create General Verifier for File Checksum commit b47549e4893a6d487c00ea0ba02619168a3f19f3 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-03T00:47:46Z Add output checksum to WordCountITOptions commit 58cd781c82fa728f34f5ab0641f8f9b6edcf449c Author: Ian Zhou <ianzhou@google.com> Date: 2016-08-05T22:31:59Z Added unit tests and error handling in removeTemporaryTables commit 36a9aa232ea56de449930194788becce585212ef Author: Thomas Groh <tgroh@google.com> Date: 2016-08-09T02:09:58Z Improve Write Error Message If provided with an Unbounded PCollection, Write will fail due to restriction of calling finalize only once. This error message fails in a deep stack trace based on it not being possible to apply a GroupByKey. Instead, throw immediately on application with a specific error message. commit d5641553cebb02f08ca7c1fe667948d39cb3962c Author: Thomas Groh <tgroh@google.com> Date: 2016-08-09T17:47:09Z Remove Streaming Write Overrides in DataflowRunner These writes should be forbidden based on the boundedness of the input PCollection. As Write explicitly forbids the application of the transform to an Unbounded PCollection, this will be equivalent in most cases; In cases where the input PCollection is Bounded, due to an UnboundedReadFromBoundedSource, the write will function as expected and does not need to be forbidden. commit 011bea9a83a828e0d8c6518ab83aa5cc4f75e146 Author: David Rieber <drieber@google.com> Date: 2016-08-09T21:05:25Z Do not add DataDisks to windmill service jobs. commit 0dfb8ff55d6f80264222fde4501ea3050d2e3911 Author: gaurav gupta <gaugupt3@cisco.com> Date: 2016-08-10T23:43:03Z Made byteArrayCoder final static commit b9f826366823003940805e6469b10df8819b0977 Author: Dan Halperin <dhalperi@google.com> Date: 2016-08-11T00:58:09Z CompressedSource: CompressedReader is never splittable The only way it's safe to split a compressed file is if the file is not compressed. This can only happen when the source itself is splittable, and that in turn will result in the inner source's reader being returned. A CompressedReader will only be created in the event that the file is NOT splittable. So remove all the logic handling splittable compressed readers, and instead go with the logic when we know/assume the file is compressed. TextIO: test compression with larger files It is important for correctness that we test with large files because otherwise the compressed file may be larger than the uncompressed file, which could mask bugs TextIOTest: flesh out more TextIOTest: add large uncompressed file commit 1d86335314685926d3eb0f9765d615a77cee75e6 Author: Thomas Groh <tgroh@google.com> Date: 2016-08-11T16:16:55Z Remove timeout in DirectRunnerTest If the test hangs due to bugs, the infrastructure should kill it. commit 37ce2a3e75cb96a4b3fdcd4938fb7fda95122724 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-11T18:26:28Z More unit test and code style fix commit 046e36eaa41659ae43866bc6fbce4f122889f286 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-11T18:55:17Z Using IOChannelUtils to resolve file path commit d056f4661da2cc399cab44c6604eaa61d1dfd178 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-14T21:51:02Z Add DoFn @Setup and @Teardown Methods annotated with these annotations are used to perform expensive setup work and clean up a DoFn after another method throws an exception or the DoFn is discarded. commit b80d96748dcb71f93697126489c924020ebbd4a9 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-15T18:27:00Z Add TransformEvaluatorFactory#cleanup This cleans up any state stored within the Transform Evaluator Factory. commit 77c90d00ae715795e73efec8f8e85e3917cf8d80 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-19T18:03:15Z Replace CloningThreadLocal with DoFnLifecycleManager This is a more focused interface that interacts with a DoFn before it is available for use and after it has completed and the reference is lost. It is required to properly support setup and teardown, as the fields in a ThreadLocal cannot all be cleaned up without additional tracking. Part of BEAM-452 . commit 39f763e16182e33019a1805d6210549934998856 Author: Pei He <peihe@google.com> Date: 2016-08-01T20:41:59Z Remove DataflowPipelineJob from examples commit 6603307062ec99639d3e3e05aebc0d1ea32ad411 Author: Thomas Groh <tgroh@google.com> Date: 2016-08-11T17:45:43Z Move ParDo Lifecycle tests to their own file These tests are not yet functional in all runners, and this makes them easier to ignore. commit d99a652f96f12bcc235caa038ffa741906336b1f Author: Maximilian Michels <mxm@apache.org> Date: 2016-08-12T15:51:02Z [flink] add missing maven config to example pom commit 424c4c492965e5a93b1c020c8d52805e3a9a9088 Author: mariusz89016 <mariusz89016@gmail.com> Date: 2016-08-13T22:35:19Z BEAM-432 Corrected BigQueryIO javadoc commit d6cf4f2a3cda9ff4ff2da105c08be8101c58e6f1 Author: Daniel Halperin <dhalperi@users.noreply.github.com> Date: 2016-08-15T22:21:15Z Exclude ParDoTest from Dataflow @RunnableOnService Until we implement it for Dataflow runner.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3636

          BEAM-79 merge gearpump-runner into master

          Follow this checklist to help us incorporate your contribution quickly and easily:

          • [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
          • [ ] Each commit in the pull request should have a meaningful subject line and body.
          • [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
          • [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
          • [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
          • [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/apache/beam gearpump-runner

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3636.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3636


          commit 9478f4117de3a2d0ea40614ed4cb801918610724
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-03-15T08:15:16Z

          BEAM-79 add Gearpump runner

          commit 02b2248a5b3c8a2c064547d7380bebc97f849bf1
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-07-20T16:07:48Z

          This closes #323

          commit 2a0ba61e8507e1539115b583749a78f14d577bd8
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-08-25T18:36:45Z

          Merge branch master into gearpump-runner

          commit 1672b5483e029292816397248dc6fe63bf51f4af
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-07-23T06:10:15Z

          move integration tests to profile

          commit 276a2e106aa1a5736666fc2eb2426b640f63cf68
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-07-28T08:30:13Z

          add package-info.java

          commit 40be715a696bb1218b209f7ad9a979b7e5d088d3
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-08-10T17:26:57Z

          Update Gearpump runner version to 0.3.0-incubating

          commit bc1b354949416db3b52c4f37c66968bdb86f0813
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2016-08-11T23:22:00Z

          Rename DoFn to OldDoFn in Gearpump runner

          commit 091a15a07c7625ae3009cefaecece3a29a34c109
          Author: Kenneth Knowles <klk@google.com>
          Date: 2016-08-25T18:40:03Z

          This closess #750

          commit fb74c936ed92c7a8548c338cc03957794fc60902
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-08-26T23:25:58Z

          gearpump: switch to stable version

          They have apparently deleted the SNAPSHOT jar and now builds are failing.

          commit bf0a2edae11416a3cbddeaff2c0a70adc272c5fe
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-08-27T00:46:42Z

          Closes #895

          commit 89921c41ca9d4c333af45efa32359a631214c1df
          Author: bchambers <bchambers@google.com>
          Date: 2016-07-29T16:41:17Z

          Remove Counter and associated code

          Aggregator is the model level concept. Counter was specific to the
          Dataflow Runner, and is now not needed as part of Beam.

          commit 7fc2c6848f002ac8b2ccbe35e6b5a576777a7af9
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-03T00:25:14Z

          BEAM-495 Create General Verifier for File Checksum

          commit b47549e4893a6d487c00ea0ba02619168a3f19f3
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-03T00:47:46Z

          Add output checksum to WordCountITOptions

          commit 58cd781c82fa728f34f5ab0641f8f9b6edcf449c
          Author: Ian Zhou <ianzhou@google.com>
          Date: 2016-08-05T22:31:59Z

          Added unit tests and error handling in removeTemporaryTables

          commit 36a9aa232ea56de449930194788becce585212ef
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-09T02:09:58Z

          Improve Write Error Message

          If provided with an Unbounded PCollection, Write will fail due to
          restriction of calling finalize only once. This error message fails in a
          deep stack trace based on it not being possible to apply a GroupByKey.
          Instead, throw immediately on application with a specific error message.

          commit d5641553cebb02f08ca7c1fe667948d39cb3962c
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-09T17:47:09Z

          Remove Streaming Write Overrides in DataflowRunner

          These writes should be forbidden based on the boundedness of the input
          PCollection. As Write explicitly forbids the application of the
          transform to an Unbounded PCollection, this will be equivalent in most
          cases; In cases where the input PCollection is Bounded, due to an
          UnboundedReadFromBoundedSource, the write will function as expected and
          does not need to be forbidden.

          commit 011bea9a83a828e0d8c6518ab83aa5cc4f75e146
          Author: David Rieber <drieber@google.com>
          Date: 2016-08-09T21:05:25Z

          Do not add DataDisks to windmill service jobs.

          commit 0dfb8ff55d6f80264222fde4501ea3050d2e3911
          Author: gaurav gupta <gaugupt3@cisco.com>
          Date: 2016-08-10T23:43:03Z

          Made byteArrayCoder final static

          commit b9f826366823003940805e6469b10df8819b0977
          Author: Dan Halperin <dhalperi@google.com>
          Date: 2016-08-11T00:58:09Z

          CompressedSource: CompressedReader is never splittable

          The only way it's safe to split a compressed file is if the file is not compressed. This can
          only happen when the source itself is splittable, and that in turn will result in the inner
          source's reader being returned. A CompressedReader will only be created in the event that
          the file is NOT splittable. So remove all the logic handling splittable compressed readers,
          and instead go with the logic when we know/assume the file is compressed.

          • TextIO: test compression with larger files

          It is important for correctness that we test with large files
          because otherwise the compressed file may be larger than the
          uncompressed file, which could mask bugs

          • TextIOTest: flesh out more
          • TextIOTest: add large uncompressed file

          commit 1d86335314685926d3eb0f9765d615a77cee75e6
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-11T16:16:55Z

          Remove timeout in DirectRunnerTest

          If the test hangs due to bugs, the infrastructure should kill it.

          commit 37ce2a3e75cb96a4b3fdcd4938fb7fda95122724
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-11T18:26:28Z

          More unit test and code style fix

          commit 046e36eaa41659ae43866bc6fbce4f122889f286
          Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
          Date: 2016-08-11T18:55:17Z

          Using IOChannelUtils to resolve file path

          commit d056f4661da2cc399cab44c6604eaa61d1dfd178
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-14T21:51:02Z

          Add DoFn @Setup and @Teardown

          Methods annotated with these annotations are used to perform expensive
          setup work and clean up a DoFn after another method throws an exception
          or the DoFn is discarded.

          commit b80d96748dcb71f93697126489c924020ebbd4a9
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-15T18:27:00Z

          Add TransformEvaluatorFactory#cleanup

          This cleans up any state stored within the Transform Evaluator Factory.

          commit 77c90d00ae715795e73efec8f8e85e3917cf8d80
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-07-19T18:03:15Z

          Replace CloningThreadLocal with DoFnLifecycleManager

          This is a more focused interface that interacts with a DoFn before it
          is available for use and after it has completed and the reference is
          lost. It is required to properly support setup and teardown, as the
          fields in a ThreadLocal cannot all be cleaned up without additional
          tracking.

          Part of BEAM-452.

          commit 39f763e16182e33019a1805d6210549934998856
          Author: Pei He <peihe@google.com>
          Date: 2016-08-01T20:41:59Z

          Remove DataflowPipelineJob from examples

          commit 6603307062ec99639d3e3e05aebc0d1ea32ad411
          Author: Thomas Groh <tgroh@google.com>
          Date: 2016-08-11T17:45:43Z

          Move ParDo Lifecycle tests to their own file

          These tests are not yet functional in all runners, and this makes them
          easier to ignore.

          commit d99a652f96f12bcc235caa038ffa741906336b1f
          Author: Maximilian Michels <mxm@apache.org>
          Date: 2016-08-12T15:51:02Z

          [flink] add missing maven config to example pom

          commit 424c4c492965e5a93b1c020c8d52805e3a9a9088
          Author: mariusz89016 <mariusz89016@gmail.com>
          Date: 2016-08-13T22:35:19Z

          BEAM-432 Corrected BigQueryIO javadoc

          commit d6cf4f2a3cda9ff4ff2da105c08be8101c58e6f1
          Author: Daniel Halperin <dhalperi@users.noreply.github.com>
          Date: 2016-08-15T22:21:15Z

          Exclude ParDoTest from Dataflow @RunnableOnService

          Until we implement it for Dataflow runner.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3636 BEAM-79 merge gearpump-runner into master Follow this checklist to help us incorporate your contribution quickly and easily: [ ] Make sure there is a [JIRA issue] ( https://issues.apache.org/jira/projects/BEAM/issues/ ) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. [ ] Each commit in the pull request should have a meaningful subject line and body. [ ] Format the pull request title like ` [BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/beam gearpump-runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3636.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3636 commit 9478f4117de3a2d0ea40614ed4cb801918610724 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-03-15T08:15:16Z BEAM-79 add Gearpump runner commit 02b2248a5b3c8a2c064547d7380bebc97f849bf1 Author: Kenneth Knowles <klk@google.com> Date: 2016-07-20T16:07:48Z This closes #323 commit 2a0ba61e8507e1539115b583749a78f14d577bd8 Author: Kenneth Knowles <klk@google.com> Date: 2016-08-25T18:36:45Z Merge branch master into gearpump-runner commit 1672b5483e029292816397248dc6fe63bf51f4af Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-07-23T06:10:15Z move integration tests to profile commit 276a2e106aa1a5736666fc2eb2426b640f63cf68 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-07-28T08:30:13Z add package-info.java commit 40be715a696bb1218b209f7ad9a979b7e5d088d3 Author: Kenneth Knowles <klk@google.com> Date: 2016-08-10T17:26:57Z Update Gearpump runner version to 0.3.0-incubating commit bc1b354949416db3b52c4f37c66968bdb86f0813 Author: manuzhang <owenzhang1990@gmail.com> Date: 2016-08-11T23:22:00Z Rename DoFn to OldDoFn in Gearpump runner commit 091a15a07c7625ae3009cefaecece3a29a34c109 Author: Kenneth Knowles <klk@google.com> Date: 2016-08-25T18:40:03Z This closess #750 commit fb74c936ed92c7a8548c338cc03957794fc60902 Author: Dan Halperin <dhalperi@google.com> Date: 2016-08-26T23:25:58Z gearpump: switch to stable version They have apparently deleted the SNAPSHOT jar and now builds are failing. commit bf0a2edae11416a3cbddeaff2c0a70adc272c5fe Author: Dan Halperin <dhalperi@google.com> Date: 2016-08-27T00:46:42Z Closes #895 commit 89921c41ca9d4c333af45efa32359a631214c1df Author: bchambers <bchambers@google.com> Date: 2016-07-29T16:41:17Z Remove Counter and associated code Aggregator is the model level concept. Counter was specific to the Dataflow Runner, and is now not needed as part of Beam. commit 7fc2c6848f002ac8b2ccbe35e6b5a576777a7af9 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-03T00:25:14Z BEAM-495 Create General Verifier for File Checksum commit b47549e4893a6d487c00ea0ba02619168a3f19f3 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-03T00:47:46Z Add output checksum to WordCountITOptions commit 58cd781c82fa728f34f5ab0641f8f9b6edcf449c Author: Ian Zhou <ianzhou@google.com> Date: 2016-08-05T22:31:59Z Added unit tests and error handling in removeTemporaryTables commit 36a9aa232ea56de449930194788becce585212ef Author: Thomas Groh <tgroh@google.com> Date: 2016-08-09T02:09:58Z Improve Write Error Message If provided with an Unbounded PCollection, Write will fail due to restriction of calling finalize only once. This error message fails in a deep stack trace based on it not being possible to apply a GroupByKey. Instead, throw immediately on application with a specific error message. commit d5641553cebb02f08ca7c1fe667948d39cb3962c Author: Thomas Groh <tgroh@google.com> Date: 2016-08-09T17:47:09Z Remove Streaming Write Overrides in DataflowRunner These writes should be forbidden based on the boundedness of the input PCollection. As Write explicitly forbids the application of the transform to an Unbounded PCollection, this will be equivalent in most cases; In cases where the input PCollection is Bounded, due to an UnboundedReadFromBoundedSource, the write will function as expected and does not need to be forbidden. commit 011bea9a83a828e0d8c6518ab83aa5cc4f75e146 Author: David Rieber <drieber@google.com> Date: 2016-08-09T21:05:25Z Do not add DataDisks to windmill service jobs. commit 0dfb8ff55d6f80264222fde4501ea3050d2e3911 Author: gaurav gupta <gaugupt3@cisco.com> Date: 2016-08-10T23:43:03Z Made byteArrayCoder final static commit b9f826366823003940805e6469b10df8819b0977 Author: Dan Halperin <dhalperi@google.com> Date: 2016-08-11T00:58:09Z CompressedSource: CompressedReader is never splittable The only way it's safe to split a compressed file is if the file is not compressed. This can only happen when the source itself is splittable, and that in turn will result in the inner source's reader being returned. A CompressedReader will only be created in the event that the file is NOT splittable. So remove all the logic handling splittable compressed readers, and instead go with the logic when we know/assume the file is compressed. TextIO: test compression with larger files It is important for correctness that we test with large files because otherwise the compressed file may be larger than the uncompressed file, which could mask bugs TextIOTest: flesh out more TextIOTest: add large uncompressed file commit 1d86335314685926d3eb0f9765d615a77cee75e6 Author: Thomas Groh <tgroh@google.com> Date: 2016-08-11T16:16:55Z Remove timeout in DirectRunnerTest If the test hangs due to bugs, the infrastructure should kill it. commit 37ce2a3e75cb96a4b3fdcd4938fb7fda95122724 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-11T18:26:28Z More unit test and code style fix commit 046e36eaa41659ae43866bc6fbce4f122889f286 Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com> Date: 2016-08-11T18:55:17Z Using IOChannelUtils to resolve file path commit d056f4661da2cc399cab44c6604eaa61d1dfd178 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-14T21:51:02Z Add DoFn @Setup and @Teardown Methods annotated with these annotations are used to perform expensive setup work and clean up a DoFn after another method throws an exception or the DoFn is discarded. commit b80d96748dcb71f93697126489c924020ebbd4a9 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-15T18:27:00Z Add TransformEvaluatorFactory#cleanup This cleans up any state stored within the Transform Evaluator Factory. commit 77c90d00ae715795e73efec8f8e85e3917cf8d80 Author: Thomas Groh <tgroh@google.com> Date: 2016-07-19T18:03:15Z Replace CloningThreadLocal with DoFnLifecycleManager This is a more focused interface that interacts with a DoFn before it is available for use and after it has completed and the reference is lost. It is required to properly support setup and teardown, as the fields in a ThreadLocal cannot all be cleaned up without additional tracking. Part of BEAM-452 . commit 39f763e16182e33019a1805d6210549934998856 Author: Pei He <peihe@google.com> Date: 2016-08-01T20:41:59Z Remove DataflowPipelineJob from examples commit 6603307062ec99639d3e3e05aebc0d1ea32ad411 Author: Thomas Groh <tgroh@google.com> Date: 2016-08-11T17:45:43Z Move ParDo Lifecycle tests to their own file These tests are not yet functional in all runners, and this makes them easier to ignore. commit d99a652f96f12bcc235caa038ffa741906336b1f Author: Maximilian Michels <mxm@apache.org> Date: 2016-08-12T15:51:02Z [flink] add missing maven config to example pom commit 424c4c492965e5a93b1c020c8d52805e3a9a9088 Author: mariusz89016 <mariusz89016@gmail.com> Date: 2016-08-13T22:35:19Z BEAM-432 Corrected BigQueryIO javadoc commit d6cf4f2a3cda9ff4ff2da105c08be8101c58e6f1 Author: Daniel Halperin <dhalperi@users.noreply.github.com> Date: 2016-08-15T22:21:15Z Exclude ParDoTest from Dataflow @RunnableOnService Until we implement it for Dataflow runner.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user manuzhang opened a pull request:

          https://github.com/apache/beam/pull/3637

          BEAM-79 Prepare for merging gearpump-runner

          Follow this checklist to help us incorporate your contribution quickly and easily:

          • [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
          • [ ] Each commit in the pull request should have a meaningful subject line and body.
          • [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
          • [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
          • [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
          • [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/manuzhang/beam gearpump-runner

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3637.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3637


          commit e655f53c2e732bcd082449ba8d0b551f417e8aaa
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-07-21T06:27:37Z

          Revert accidental changes to sdks/java/pom.xml

          commit daa7566939e212095ef745ff004580bfe4209b38
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-07-21T06:58:13Z

          Upgrade BEAM version to 2.2.0-SNAPSHOT in gearpump-runner

          commit 49d4ed59567852e978d14729c89b9ea91bb96c32
          Author: manuzhang <owenzhang1990@gmail.com>
          Date: 2017-07-21T14:33:36Z

          Add beam-runners-gearpump dependency to javadoc


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user manuzhang opened a pull request: https://github.com/apache/beam/pull/3637 BEAM-79 Prepare for merging gearpump-runner Follow this checklist to help us incorporate your contribution quickly and easily: [ ] Make sure there is a [JIRA issue] ( https://issues.apache.org/jira/projects/BEAM/issues/ ) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. [ ] Each commit in the pull request should have a meaningful subject line and body. [ ] Format the pull request title like ` [BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/beam gearpump-runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3637.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3637 commit e655f53c2e732bcd082449ba8d0b551f417e8aaa Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-07-21T06:27:37Z Revert accidental changes to sdks/java/pom.xml commit daa7566939e212095ef745ff004580bfe4209b38 Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-07-21T06:58:13Z Upgrade BEAM version to 2.2.0-SNAPSHOT in gearpump-runner commit 49d4ed59567852e978d14729c89b9ea91bb96c32 Author: manuzhang <owenzhang1990@gmail.com> Date: 2017-07-21T14:33:36Z Add beam-runners-gearpump dependency to javadoc
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user manuzhang closed the pull request at:

          https://github.com/apache/beam/pull/3611

          Show
          githubbot ASF GitHub Bot added a comment - Github user manuzhang closed the pull request at: https://github.com/apache/beam/pull/3611
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3637

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3637
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3636

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3636
          Hide
          mauzhang Manu Zhang added a comment -

          Hi Tyler Akidau,

          We've reached a milestone after merging gearpump-runner into master. Future development will be under new JIRAs. Can we close this issue now ?

          Thanks for asking on Twitter and initiating this work a year ago.

          Manu

          Show
          mauzhang Manu Zhang added a comment - Hi Tyler Akidau , We've reached a milestone after merging gearpump-runner into master. Future development will be under new JIRAs. Can we close this issue now ? Thanks for asking on Twitter and initiating this work a year ago. Manu
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user huafengw opened a pull request:

          https://github.com/apache/beam/pull/3726

          BEAM-79 Add Gearpump runner profile to Java8 Archetype POM

          Follow this checklist to help us incorporate your contribution quickly and easily:

          • [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
          • [ ] Each commit in the pull request should have a meaningful subject line and body.
          • [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
          • [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
          • [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
          • [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/huafengw/beam example

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/beam/pull/3726.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3726


          commit 31810c459ea95cb7816261087a7e73dc0e0338c3
          Author: huafengw <fvunicorn@gmail.com>
          Date: 2017-08-17T03:58:33Z

          BEAM-79 Add Gearpump runner profile to Java8 Archetype POM


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user huafengw opened a pull request: https://github.com/apache/beam/pull/3726 BEAM-79 Add Gearpump runner profile to Java8 Archetype POM Follow this checklist to help us incorporate your contribution quickly and easily: [ ] Make sure there is a [JIRA issue] ( https://issues.apache.org/jira/projects/BEAM/issues/ ) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. [ ] Each commit in the pull request should have a meaningful subject line and body. [ ] Format the pull request title like ` [BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement] ( https://www.apache.org/licenses/icla.pdf ). — You can merge this pull request into a Git repository by running: $ git pull https://github.com/huafengw/beam example Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3726.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3726 commit 31810c459ea95cb7816261087a7e73dc0e0338c3 Author: huafengw <fvunicorn@gmail.com> Date: 2017-08-17T03:58:33Z BEAM-79 Add Gearpump runner profile to Java8 Archetype POM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/beam/pull/3726

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/3726

            People

            • Assignee:
              mauzhang Manu Zhang
              Reporter:
              takidau Tyler Akidau
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Development