This is manuzhang from Gearpump team. May I take this task ?

Happy to assign it to you, as you will clearly be the expert on Gearpump

But please note that Beam is still very much under construction and there are a number of breaking API changes likely in the near future. So please reach out before getting beyond the early design phase / determining how well the models align. If you haven't yet, I'd start with these resources: http://beam.incubator.apache.org/getting_started/

Thanks. I'm glad to take this task, and the Gearpump team is also very excited. I'll follow the guides and reach out for you guys whenever in doubts.

Apache Beam is both batch and streaming process and Gearpump is focus on streaming processing .

That's fine. Batch is essentially a subset of streaming from a semantics perspective. And even if it weren't, we don't require all runners to implement all of the features in the model; runners are allowed to implement only the pieces that make sense for them.

Here is a design document . Please kindly comment it.

It sounds good to me. Thanks Manu !

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/323

BEAM-79 add Gearpump runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

This PR adds Gearpump runner to Beam meeting the goals of phase 1 in the [design document](https://docs.google.com/document/d/1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g/edit).

The Gearpump runner supports the following functionalities,

• Transformations: ParDo, GroupByKey, Flatten
• Windows: using Beam's window logic and code
• side outputs
• serialization/deserialization: using Gearpump's Kryo serializer
• sources: Beam's UnboundedSource
• message delivery guarantee: at-most-once
• tests: integration test for various translators

Here's a snapshot of running the following Beam example on Gearpump cluster

```java
PCollection<KV<String, Long>> wordCounts =
p.apply(Read.from(new UnboundedTextSource()).named("WordStream"))
.apply(ParDo.of(new ExtractWordsFn()))
.apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(10))))
.apply(Count.<String>perElement());

wordCounts.apply(ParDo.of(new FormatAsStringFn()));
```


Note that the Gearpump runner is still in early stage and lacking capabilities like trigger, side inputs, aggregator. However, I'd like to have the community to get a feel of what Gearpump is like, whether Beam and Gearpump go well, and gather ideas for improvements.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #323

commit 73e5978f599bcf32ed8c2f1d54b6bd3bd8350092
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-03-15T08:15:16Z

BEAM-79 add Gearpump runner

Starting to review and test. Thanks !

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/323

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/714

BEAM-79 Add Gearpump runner to runners parent pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [x] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [x] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [x] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

I think this line got lost in the shuffle. It is why we aren't seeing the module in the Jenkins web UI, though the [tests are running and passing](https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/).

R: @dhalperi
CC: @manuzhang

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam gearpump-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #714

commit f5e1abb61ea36866cd2d15be6e82c2b0785a4d9c
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-22T19:50:43Z

Add Gearpump runner to runners parent pom.xml

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/714

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/750

BEAM-79 Merge branch 'master' into gearpump_runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump_runner_sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #750

commit 34467f92d5a31b47f95b734c737fde0a8277311b
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-15T17:51:24Z

Make TransformEvaluatorFactory reuse Explicit

Transform Evaluator Factories must be reused for the entire execution of
a Pipeline and must not be reused across pipelines.

Remove EvaluatorKey, and key explicitly by the transform application.

commit ad6ee01825740ee47f25ca036fa5f352375bbe6e
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-19T06:40:13Z

Checkstyle: enforce package-info.java presence

Ignores tests and examples

commit c4ad11832235eef0b73d299cd02d1a224c130ece
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-20T16:16:28Z

Closes #692

commit 6d7efe3df8cde590722d94441eca3922a3a67734
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-20T17:30:55Z

This closes #666

commit 5f19e4caf207ff02d226fec2933be2f37ca66b4d
Author: Thomas Groh <tgroh@google.com>
Date: 2016-06-30T17:06:52Z

Add withAllowedLateness with Closing Behavior to Window

This makes the static constructors for withAllowedLateness symmetric to
the PTransform builder methods. It also allows references to
Window#withAllowedLateness(Duration, ClosingBehavior).

commit f547f70e1c9535ff663d124f67c72c7ec2c55e9e
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-20T17:40:13Z

This closes #567

commit 00195d2543eb347cc3669a4ac89e98da0bc4dca4
Author: Thomas Groh <tgroh@google.com>
Date: 2016-06-28T22:44:49Z

Use the ParDo Application to Cache DoFns

A DoFn application is the scope of reuse.

Factor CloningThreadLocal as the top-level class instead of
SerializableCloningThreadLocalCacheLoader, and extract the Fn from the
AppliedPTransform when loading an absent element.

commit 436e4a34ebb222545cb03cb6d39ea4ca2d905254
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-20T17:55:53Z

This closes #554

commit b240525affb205a83054577233f3a4a508fe1c72
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-18T19:05:02Z

BigQueryIO: move to google-cloud-platform module

• Move package from io to io.gcp.bigquery
• Move from SDK core into GCP-IO module
• Fixup references and import orders
• Separate AvroUtils into generic AvroUtils and BigQueryAvroUtils
• Rewrite a unit test in sdk core to not depend on BigQueryIO
• Fixup Javadoc in SDK core that need not depend on BigQueryIO
• Make utility classes package-private

commit 7ec8781a2e18548a23c882329f0b50f7254202ec
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-20T20:02:43Z

Closes #681

commit bdb65278873a5010a625dc6a569ba25b17374c06
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-20T04:36:10Z

Add os-maven-plugin to Spark runner

commit 84332ee9716233af928e85c14c534714ab828531
Author: Chandni Singh <chandni.singh@capitalone.com>
Date: 2016-07-20T00:30:16Z

BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes

commit 6d5e8186a2da532eb1c29097bc1259a19d9f72c9
Author: Luke Cwik <lcwik@visitor-lcwik.wat.corp.google.com>
Date: 2016-07-21T13:34:52Z

BEAM-372 added a test that verifies if a coder consumes bytes equal to encoded bytes

This closes #695

commit ae2144196c351cc2ee544e030d793929d8607696
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-20T17:26:02Z

BigtableIO: upgrade to 0.9.1

• Use the uber jar
• Remove OS classifier mumbo jumbo
• Move common dependency versioning to root pom

commit 0eb81fdc15bc3dce28e24e73427bd389e0f6d60d
Author: Luke Cwik <lcwik@visitor-lcwik.wat.corp.google.com>
Date: 2016-07-21T15:26:41Z

BEAM-445 BigtableIO: upgrade to 0.9.1

This closes #701

commit 232c0e9610a5d5559598a2059577b699299a0b76
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-20T16:49:47Z

Rename DirectRegistrar Registrars

This removes the duplication of "DirectRunner" and "DirectOptions"
classes.

commit 9cdaed0d8a401caf5a9efb626fa629fd6f4f4627
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-21T17:24:54Z

Closes #700

commit 7337ecf147ca69262978db116a5a1f739aca28ae
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2016-05-31T16:04:52Z

BEAM-316 Add file scheme support in TextIO

commit 38061d3531a1ca13ef825a6ee50872e420ab1494
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-18T22:35:06Z

BEAM-316 Add file URI handling

• Register FileIOChannelFactory for file scheme
• Modify FileIOChannelFactory to dynamically remove the file:// scheme string.

commit adb472083460bc175a0685133eadb6a336f1a4a2
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-22T03:17:54Z

Closes #684 and #402

commit 060bc802f0bff8ca1a076e9f93a623058e1d651b
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-22T04:01:36Z

Execute RunnableOnService tests only when runner options provided

Previously, the situation was this:

• All runners inherit a RunnableOnService integration-test
  execution referencing runnableOnServicePipelineOptions
  whether or not the variable was set. Basically an unbound
  variable reference.
• The Dataflow runner had a profile disabling it if
  runnableOnServicePipelineOptions was not set.
• Before they got configured, Flink and Spark had to
  do extra work to explicitly prevent the invalid
  configuration from being used.

After this change:

• All runners inherit the same integration-test execution
  but only if the variable it requires is present.
• Dataflow doesn't have any special profile.
• Flink and Spark are unchanged, since they do set
  up the variable themselves. When they move to running
  only as postcommit, like Dataflow does, the hardcoding
  is expected to either move to a profile or move to
  the Jenkins invocation.

commit 07a313f1af78349040afd9ff6b01e60cdcd567a3
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-22T07:39:59Z

Closes #709

commit 75df931a55dbd3f93d9143574cdad7291d3457aa
Author: Jesse Anderson <jesse@smokinghand.com>
Date: 2016-07-22T20:54:26Z

Added BigInteger to TypeDescriptors class.

commit 12b60ffa5d18812674299a97e84af31d063e170c
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-22T23:51:37Z

Closes #716

commit bf3af5de9e6f8126ffd2ccd3f7a68a84e55e90ff
Author: Pei He <peihe@google.com>
Date: 2016-07-22T22:03:38Z

BEAM-480 Move insertAll() from BigQueryTableInserter to BigQueryServices

commit 122cd04663529f4fc44530ed9e144e0dfd68360f
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-22T23:52:50Z

Closes #717

commit 12fb77811d7bf8c72a3c0a48f391bab3c815fbef
Author: Sela <ansela@paypal.com>
Date: 2016-06-27T17:59:07Z

Remove dependency on beam-examples-java.

Duplicate WordCount into spark examlpes package.

Duplicate parts of TfIdf from beam examlpes.

Better reuse of WordCount and its parts.

Remove dependency on beam-examples-java

commit d3dc368666d5b700c5026fe08455e1210692ce3e
Author: Sela <ansela@paypal.com>
Date: 2016-07-04T12:33:57Z

Remove JavaDoc links to beam-examples.

commit cf1464465e1e65be2e4896cc6f693f8bbe82b844
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-07-23T02:51:50Z

Closes #539

commit 2fe387707d1e115b578f5ee643bb99c0e4667ee0
Author: Maximilian Michels <mxm@apache.org>
Date: 2016-07-20T14:06:06Z

[flink] improve example section in README

• updates the README
• repairs broken exec configuration

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/750

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/943

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/956

BEAM-79 fix integration-test failure

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/956.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #956

commit 272fe9f6eb6c721a1d4a39cc816e3de9dc705373
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-09-14T03:20:27Z

BEAM-79 fix integration-test failure

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/956

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/959

BEAM-79 upgrade surefire version to exclude tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #959

commit c879a7fa02259458029d8dca82ecaf80d36019b7
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-09-14T22:52:09Z

BEAM-79 upgrade surefire version to exclude tests

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/959

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1193

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1234

BEAM-79 port OldDoFn to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1234.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1234

commit ccee98f39236062dd501b63d9a89e88dd1fd47bd
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-10-31T03:52:22Z

BEAM-79 port OldDoFn to new DoFn

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/1234

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1306

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1663

BEAM-79 merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1663

commit 0af145bba721f7cdf3fbcd15330df433d5a8431d
Author: Davor Bonaci <davor@google.com>
Date: 2016-12-07T01:00:31Z

This closes #1504

commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
Author: Jakob Homan <jghoman@gmail.com>
Date: 2016-12-07T00:59:50Z

BEAM-1099 Minor typos in KafkaIO

commit afedd68e806830549724dfc0f2565d756cc6b46d
Author: Davor Bonaci <davor@google.com>
Date: 2016-12-07T01:03:54Z

This closes #1524

commit e8c9686a2e898d38afd692328eb171c542084747
Author: Pei He <peihe@google.com>
Date: 2016-11-23T23:59:56Z

BEAM-1047 Add DataflowClient wrapper on top of JSON library.

commit ded58832ceaef487f4590d9396f09744288c955d
Author: Pei He <peihe@google.com>
Date: 2016-11-24T00:14:27Z

[Code Health] Remove redundant projectId from DataflowPipelineJob.

commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
Author: Pei He <peihe@google.com>
Date: 2016-11-28T19:47:42Z

BEAM-1047 Update dataflow runner code to use DataflowClient wrapper.

commit b2b570f27808b1671bf6cd0fc60f874da671d4ca
Author: bchambers <bchambers@google.com>
Date: 2016-12-07T01:08:13Z

Closes #1434

commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152
Author: Sam McVeety <sgmc@google.com>
Date: 2016-10-30T18:58:44Z

BigQueryIO.Read: support runtime options

commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-07T05:28:13Z

FLINK-1102 Fix Aggregator Registration in Flink Batch Runner

commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-07T07:26:02Z

This closes #1530

commit baf5e6bd9b1011f4c5c3974aa46393471b340c15
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2016-12-07T07:37:33Z

BEAM-1094 Set test scope for Kafka IO and junit

commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f
Author: Sela <ansela@paypal.com>
Date: 2016-12-07T08:31:38Z

This closes #1531

commit dce3a196a3a26fdd42225520faf3d9084ee48123
Author: Sela <ansela@paypal.com>
Date: 2016-12-07T09:20:07Z

BEAM-329 Update Spark runner README.

commit 02bb8c375c48847b1686d70184fb194500a62e8c
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2016-12-07T11:51:09Z

BEAM-329 This closes #1532

commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-06T23:20:28Z

Port most of DoFnRunner Javadoc to new DoFn

commit 1526184ae8be1f8ae6863f830653204157a584cd
Author: Thomas Groh <tgroh@google.com>
Date: 2016-12-07T16:51:02Z

This closes #1527

commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T05:01:37Z

Port ParDoTest from OldDoFn to new DoFn

commit ae52ec1bc3f3120e9f8e150e50c18564055a467c
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T17:00:18Z

This closes #1529

commit 55d333bff68809ff1a9154491ace02d2d16e3b85
Author: Thomas Groh <tgroh@google.com>
Date: 2016-12-05T22:29:05Z

Only provide expanded Inputs and Outputs

This removes PInput and POutput from the immediate API Surface of
TransformHierarchy.Node, and forces Pipeline Visitors to access only
the expanded version of the output.

This is part of the move towards the runner-agnostic representation of a
graph.

commit 5b31a369962907e257de8019fbf6cde4c615b1c0
Author: Thomas Groh <tgroh@google.com>
Date: 2016-12-07T17:14:38Z

This closes #1511

commit 43fef2775145f67def3ab8a246ecca192a7d650b
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-07T12:06:57Z

BEAM-905 Add shading config to examples archetype and enable it for Flink

This makes the Flink quickstart work out of the box.

commit b44a7ac4a2a0ac9fd7a8e38cc5eea5073c629085
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-07T22:55:03Z

Closes #1533

commit 6439f701d1008d6a0432828e11e0fcc8a4fe6ecc
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-07T23:40:58Z

BEAM-1108 DataflowRunner: remove deprecated TEARDOWN_POLICY control

commit 3b2e0290ddbedd199926a70f36c02ea4515841cb
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-08T01:18:11Z

Closes #1546

commit 1b1252074dd6b57f4fb88ceb82c704d3d3d8147f
Author: Alexey Diomin <diominay@gmail.com>
Date: 2016-12-07T05:39:27Z

BEAM-1095 Add support set config for reuse-object on flink

commit c53e0b1623e3ee3c08c329e2716440f031681591
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-08T01:44:07Z

This closes #1518

commit f1a5704a505b01d7d4649b61d1f6697859367964
Author: Alexey Diomin <diominay@gmail.com>
Date: 2016-12-07T05:48:35Z

BEAM-1096 Flink streaming side output optimization using SplitStream

commit 6807480a97f2315b3f48ad8dd5accb4e30475fa4
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-08T01:55:32Z

This closes #1520

commit 3f8c80769a3bb38da64c6628fd8c4669fcac794b
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T04:10:06Z

Add DoFn.OnTimerContext

commit 2883062eebe8dba849ab89627f6aeb53266ac1a8
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T04:10:21Z

Access to OnTimerContext via DoFnInvokers.ArgumentProvider

Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/1663

GitHub user manuzhang reopened a pull request:

https://github.com/apache/incubator-beam/pull/1663

BEAM-79 merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [x] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [x] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [x] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1663

commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
Author: Jakob Homan <jghoman@gmail.com>
Date: 2016-12-07T00:59:50Z

BEAM-1099 Minor typos in KafkaIO

commit afedd68e806830549724dfc0f2565d756cc6b46d
Author: Davor Bonaci <davor@google.com>
Date: 2016-12-07T01:03:54Z

This closes #1524

commit e8c9686a2e898d38afd692328eb171c542084747
Author: Pei He <peihe@google.com>
Date: 2016-11-23T23:59:56Z

BEAM-1047 Add DataflowClient wrapper on top of JSON library.

commit ded58832ceaef487f4590d9396f09744288c955d
Author: Pei He <peihe@google.com>
Date: 2016-11-24T00:14:27Z

[Code Health] Remove redundant projectId from DataflowPipelineJob.

commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
Author: Pei He <peihe@google.com>
Date: 2016-11-28T19:47:42Z

BEAM-1047 Update dataflow runner code to use DataflowClient wrapper.

commit b2b570f27808b1671bf6cd0fc60f874da671d4ca
Author: bchambers <bchambers@google.com>
Date: 2016-12-07T01:08:13Z

Closes #1434

commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152
Author: Sam McVeety <sgmc@google.com>
Date: 2016-10-30T18:58:44Z

BigQueryIO.Read: support runtime options

commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-07T05:28:13Z

FLINK-1102 Fix Aggregator Registration in Flink Batch Runner

commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-07T07:26:02Z

This closes #1530

commit baf5e6bd9b1011f4c5c3974aa46393471b340c15
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2016-12-07T07:37:33Z

BEAM-1094 Set test scope for Kafka IO and junit

commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f
Author: Sela <ansela@paypal.com>
Date: 2016-12-07T08:31:38Z

This closes #1531

commit dce3a196a3a26fdd42225520faf3d9084ee48123
Author: Sela <ansela@paypal.com>
Date: 2016-12-07T09:20:07Z

BEAM-329 Update Spark runner README.

commit 02bb8c375c48847b1686d70184fb194500a62e8c
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2016-12-07T11:51:09Z

BEAM-329 This closes #1532

commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-06T23:20:28Z

Port most of DoFnRunner Javadoc to new DoFn

commit 1526184ae8be1f8ae6863f830653204157a584cd
Author: Thomas Groh <tgroh@google.com>
Date: 2016-12-07T16:51:02Z

This closes #1527

commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T05:01:37Z

Port ParDoTest from OldDoFn to new DoFn

commit ae52ec1bc3f3120e9f8e150e50c18564055a467c
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T17:00:18Z

This closes #1529

commit 55d333bff68809ff1a9154491ace02d2d16e3b85
Author: Thomas Groh <tgroh@google.com>
Date: 2016-12-05T22:29:05Z

Only provide expanded Inputs and Outputs

This removes PInput and POutput from the immediate API Surface of
TransformHierarchy.Node, and forces Pipeline Visitors to access only
the expanded version of the output.

This is part of the move towards the runner-agnostic representation of a
graph.

commit 5b31a369962907e257de8019fbf6cde4c615b1c0
Author: Thomas Groh <tgroh@google.com>
Date: 2016-12-07T17:14:38Z

This closes #1511

commit 43fef2775145f67def3ab8a246ecca192a7d650b
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-07T12:06:57Z

BEAM-905 Add shading config to examples archetype and enable it for Flink

This makes the Flink quickstart work out of the box.

commit b44a7ac4a2a0ac9fd7a8e38cc5eea5073c629085
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-07T22:55:03Z

Closes #1533

commit 6439f701d1008d6a0432828e11e0fcc8a4fe6ecc
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-07T23:40:58Z

BEAM-1108 DataflowRunner: remove deprecated TEARDOWN_POLICY control

commit 3b2e0290ddbedd199926a70f36c02ea4515841cb
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-12-08T01:18:11Z

Closes #1546

commit 1b1252074dd6b57f4fb88ceb82c704d3d3d8147f
Author: Alexey Diomin <diominay@gmail.com>
Date: 2016-12-07T05:39:27Z

BEAM-1095 Add support set config for reuse-object on flink

commit c53e0b1623e3ee3c08c329e2716440f031681591
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-08T01:44:07Z

This closes #1518

commit f1a5704a505b01d7d4649b61d1f6697859367964
Author: Alexey Diomin <diominay@gmail.com>
Date: 2016-12-07T05:48:35Z

BEAM-1096 Flink streaming side output optimization using SplitStream

commit 6807480a97f2315b3f48ad8dd5accb4e30475fa4
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2016-12-08T01:55:32Z

This closes #1520

commit 3f8c80769a3bb38da64c6628fd8c4669fcac794b
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T04:10:06Z

Add DoFn.OnTimerContext

commit 2883062eebe8dba849ab89627f6aeb53266ac1a8
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T04:10:21Z

Access to OnTimerContext via DoFnInvokers.ArgumentProvider

commit 42b506f06dbd73e03a2cfad4e7677e9698b3c020
Author: Kenneth Knowles <klk@google.com>
Date: 2016-12-07T04:18:18Z

Add OnTimerContext parameter support to DoFnSignature

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1663

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/1828

BEAM-79 Fix PostCommit test confs for Gearpump runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-post-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1828.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1828

commit 4fd216b6b9f7427a7df118a60202752ebeb1836d
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-01-24T05:16:07Z

BEAM-79 Fix PostCommit test confs for Gearpump runner

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1828

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/1935

BEAM-79 Fix Gearpump runner for WindowingTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-window

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1935

commit 82994256798fe85c4e2fbbab8262836c51ae1ab8
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-02-07T14:14:18Z

BEAM-79 Fix Gearpump runner for WindowingTest

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1935

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/2150

BEAM-79 Add SideInput support for GearpumpRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2150

commit 08b18bf326a7ac1bae485fbdf73aeab0de46c716
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-02-14T04:33:31Z

BEAM-79 Add SideInput support for GearpumpRunner

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2150

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2241

GitHub user huafengw opened a pull request:

https://github.com/apache/beam/pull/2607

BEAM-79 Upgrade Gearpump to a stable version

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huafengw/beam upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2607.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2607

commit 3ba9b229a0a2cfcd84d5b83f208748c060caf20d
Author: huafengw <fvunicorn@gmail.com>
Date: 2017-04-20T01:36:45Z

BEAM-79 Upgrade Gearpump to a stable version

Github user huafengw closed the pull request at:

https://github.com/apache/beam/pull/2607

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/2888

BEAM-79 Merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2888.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2888

commit d9595d35176d2572423f6be069982b05eea204dd
Author: Maria Garcia Herrero <mariagh@google.com>
Date: 2017-04-26T23:22:09Z

Add extra message to deprecated annotation

commit 42e3a6f8588046f257e5c780dec12d16d0314a66
Author: Ahmet Altay <altay@google.com>
Date: 2017-04-27T18:25:18Z

This closes #2719

commit 9174ebf48391f393761b2ce085d2f165ebb2c7bb
Author: Tibor Kiss <tibor.kiss@gmail.com>
Date: 2017-04-27T12:12:13Z

BEAM-1695 Improve Python-SDK's programming guide

commit 884935cb98f0e29bd965fda9182970423d111d7e
Author: Ahmet Altay <altay@google.com>
Date: 2017-04-27T19:03:22Z

This closes #2738

commit e047b69efab9c988011303cf2eda86ac408b38c2
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T15:30:47Z

BEAM-2096 Make DataflowMetrics more resilient

DataflowMetrics seems to have many hard-coded assumptions about what will be
returned by the Dataflow service that, which will likely break when users
use new types of metrics or if the Dataflow service makes minor adjustments
in our it sends metrics back to the user.

In order for code to continue working in these cases, handle errors by logging
and skipping.

commit fdf2de9994268f5e1d704aa30a4504ac6ee624b8
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T20:10:32Z

This closes #2739

commit f93a27755921e73141fd97de042e6dcd04e10a47
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-26T00:29:02Z

BEAM-59 Add FileSystems#matchNewResource

The new FileSystems API needs a way to generate a ResourceId for a
resource that does not exist. This does not come up in sources, because
we typically are just matching existing files. However, sinks need the
ability to reference a new directory (say, in order to create it).

Couldn't think of anything better than a simple function that says
"treat this string as a full resource path with the specified type",
which is what FileSystems#matchNewResource is.

commit 14a90d7c5196d96de66e897687d28202d790439f
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T21:56:24Z

This closes #2728

commit d3b399fa84bff45b0960368d32d4c69e433f4c94
Author: Thomas Groh <tgroh@google.com>
Date: 2017-04-26T22:12:54Z

Remove Coder Encoding IDs

These were never safe to use, and will not be used once Coders have been
migrated to the Runner API representation.

commit c4936958834aa55f0f1fcb73fe9b8419c4c9eb9b
Author: Thomas Groh <tgroh@google.com>
Date: 2017-04-27T22:22:55Z

This closes #2717

commit 1925a50d860b7b8f8422f1c2f251d0ea11def736
Author: Aviem Zur <aviemzur@gmail.com>
Date: 2017-04-27T20:48:02Z

BEAM-2029 NullPointerException when using multi output ParDo in Spark runner in streaming mode.

commit a46eb1a80074dc9c382da4d71725ffc61f2e65ba
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T22:33:39Z

This closes #2746

commit 9a3b830c8737859c6afaa6e6e1922b037a78741f
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T22:35:10Z

BEAM-59 LocalResources: helper utility to create LocalResourceId

Primarily for testing.

commit f71395bfd8f94cbe8d78098f5bb66a1dce65e74b
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T23:30:50Z

This closes #2751

commit 95fdb8f4dbb6bb8ec6d80152aa85cc970ab13366
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-27T20:51:53Z

WriteFiles: trivial message fix

commit 674ae9a475f1f236230697d5cb1c33d36fec49db
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-28T00:20:54Z

This closes #2755

commit 06f4ef40bec7d6d3c5826fb9e0c164eb9c05efe7
Author: Ahmet Altay <altay@google.com>
Date: 2017-04-28T00:14:41Z

Add a simple snippet for a composite transform.

commit 948f7ef2fc37b30809cf1adb2631eeb6a1e5eff3
Author: Ahmet Altay <altay@google.com>
Date: 2017-04-28T00:35:37Z

This closes #2756

commit e1d5174eaffef3b2ac7df6187c43ffbccce32b09
Author: Luke Cwik <lcwik@google.com>
Date: 2017-04-27T23:24:57Z

BEAM-2031 Add support to PipelineOptionsFactory/ProxyInvocationHandler for auto registration of Jackson modules

This is towards adding a mixin module for Hadoop Configuration so that it can live on a PipelineOptions object.

commit 185dc47982078124abf6e851ad1c66ebc4ae8c54
Author: Lukasz Cwik <lcwik@google.com>
Date: 2017-04-28T03:15:40Z

BEAM-2031 Add support to PipelineOptionsFactory/ProxyInvocationHandler for auto registration of Jackson modules

This closes #2752

commit 34c3ee7bfe0e0b6b95d0e9ec421a061116fc19b0
Author: Vikas Kedigehalli <vikasrk@google.com>
Date: 2017-04-27T21:58:24Z

BEAM-59 Remove IOChannelFactory usage from NumberedShardedFile/ExplicitShardedFile

commit 47aaf11250e61d61c0bb3616063dbd5b7f2adb88
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-04-28T06:03:12Z

This closes #2749

commit f37d954873ae03cdae854837570b62e334be9022
Author: Thomas Weise <thw@apache.org>
Date: 2017-04-27T09:06:50Z

BEAM-1981 Fix ApexTimerInternals serialization error.

commit 40a924d8d4e06ecbd7252ddbd6558415f899c123
Author: Luke Cwik <lcwik@google.com>
Date: 2017-04-27T18:34:01Z

BEAM-1871 Remove irrelevant runtime dependencies.

Confirmed that sdks/java/io/gcp uses 1.22 for the removed libraries.

commit 849e39dfcf28a6edbf3f4c0469229778e0a52cb3
Author: Luke Cwik <lcwik@google.com>
Date: 2017-04-28T15:14:46Z

BEAM-1871 Remove irrelevant runtime dependencies

This closes #2748

commit ce88c88b14e963ac17fac83dd19495c835c1f6cb
Author: Luke Cwik <lcwik@google.com>
Date: 2017-04-28T01:13:43Z

BEAM-2005 Move getScheme from FileSystemRegistrar to FileSystem

Note that I needed to update FileSystems to instantiate the FileSystem(s) upfront instead of remembering the mapping from scheme to registrar.

commit ddabab3faded59e782455486fedf74b2ff60b60c
Author: Luke Cwik <lcwik@google.com>
Date: 2017-04-28T16:00:26Z

BEAM-2005 Move getScheme from FileSystemRegistrar to FileSystem

This closes #2757

commit fa422b9a81c303616786e8198d22c70df983d28e
Author: Ismaël Mejía <iemejia@apache.org>
Date: 2017-04-28T15:25:30Z

Remove missing references to the flink examples package after refactor

commit d08b72892bf900995667a37239e610bc2cd960d0
Author: Ismaël Mejía <iemejia@apache.org>
Date: 2017-04-28T16:26:08Z

This closes #2760

commit 594705d717dd6e5b81c08dcf4c56f8f6e7422737
Author: Peter Gergo Barna <pbarna@hortonworks.com>
Date: 2017-04-27T12:37:26Z

BEAM-2094 WordCount examples produce garbage for non-English input text

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2888

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3172

BEAM-79 Sync gearpump-runner with master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3172

commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-06T00:16:34Z

BEAM-2190 pom.xml: do a better job of dependency management

Even if Beam appears to have the correct dependencies, we cannot
guarantee that modules that depend on us transitively get the right
dependencies. For example, even though grpc-protobuf-lite has
protobuf-lite excluded, and the Maven Enforcer banned-dependencies
check passes... if a user happens to get a transitive dependency on
grpc-all first, they may pull in grpc-protobuf from that other source
without the exclusion. Thus we need to exclude protobuf-lite from
grpc-all as well.

While we're here, also add guava-jdk5 to the set of banned dependencies,
though (as above) we cannot currently properly identify the places it
might be transitively exposed in a users' pom.xml.

commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-08T18:35:56Z

This closes #2934

commit 5bac40efcc7bcd9d53183579ec6089c23aeda782
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-06T02:06:03Z

BEAM-2212 FileBasedSource: improve message when logging.

ValueProvider should not be printed, rather the string instead.

commit 9423babd8f827e843723c218441e9a91aaa7b361
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-08T16:59:16Z

BEAM-2212 FileBasedSource: refactor to remove uses of fileOrPatternSpec.get()

Makes it less likely to have errors from printing ValueProviders instead of runtime values

commit e1791c3f83a007013625b43f9976e7ffa64e8a49
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-08T18:48:42Z

This closes #2956

commit 13e51c9cd114eec73c47b71f46214dd92ee81048
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-06T02:49:29Z

Convert Executor Services to use Daemon Threads

This will cause the DirectRunner to automatically shut down when the
worker threads are shut down.

commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c
Author: Thomas Groh <tgroh@google.com>
Date: 2017-05-08T19:52:09Z

This closes #2954

commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80
Author: Sourabh Bajaj <sourabhbajaj@google.com>
Date: 2017-05-07T20:55:20Z

BEAM-2206 Move pipelineOptions into options modules

commit 604c2f59e2a928a95106814b0b55b8097dc39faa
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-08T19:56:15Z

This closes #2925

commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18
Author: Reuven Lax <relax@google.com>
Date: 2017-05-08T16:06:55Z

TableDescription is allowed to be null.

commit c481396edc7a3e906cd79dcc2a356baf9c93895e
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-08T20:41:01Z

This closes #2953

commit 1fd15bd7441e1be72feb3f9535ffe536d730a531
Author: Reuven Lax <relax@google.com>
Date: 2017-05-07T18:50:08Z

Shortens PubsubIO names for reading/writing messages.

commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-08T20:45:25Z

This closes #2950

commit b7635aadc2532b5c3bdc38f070865091ebf901c1
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-08T20:42:02Z

Remove apex README.md. This information is already in the website in
quickstart and apex runner pages.
https://github.com/apache/beam-site/pull/232 moves small bits of missing
content.

commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-08T21:28:36Z

This closes #2962

commit fba4c010ce90db8b5cdac489d47126ca3ee8db65
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-08T22:00:33Z

Remove readme.md files that are covered in web site

commit 152c5bcc7f6fb53565ed132653384f4244542b59
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-08T22:04:46Z

This closes #2966

commit 78c29238aa471138fb55681d8700b095a3606e0d
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-05-08T21:54:17Z

BEAM-2224 Fix temp file management on Windows.

commit f315cc9ff649ccee9c9aff18f4af30cfbd58e5e5
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-05-08T23:49:05Z

Closes #2965

commit fd1dda09fc4db48cfbc21afd6310320533fffe78
Author: Reuven Lax <relax@google.com>
Date: 2017-04-29T14:33:54Z

BEAM-2154 More scalable dynamic BigQueryIO.Write

If too many tables are generated in a bundle, spill and group the results
before writing files. Generating hundreds or thousands of file write buffers
in a single bundle was causing workers to crash with out of memory.

commit 6a2586a4ee279f66fd4eae816e93fa8e397acf6b
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-09T01:02:14Z

This closes #2883

commit 367fcb28d544934797d25cb34d54136b2d7d6e99
Author: Vikas Kedigehalli <vikasrk@google.com>
Date: 2017-05-06T02:24:51Z

Remove google api BackOff usage from sdks/core

commit 87578a6a69f15bca502d9b4c15999aa383e539a5
Author: Vikas Kedigehalli <vikasrk@google.com>
Date: 2017-05-06T02:33:45Z

Use guava Base64 encoding instead of google api client

commit 88e044fda0a7d0ea1372f075a4c7a4bd54af47da
Author: Luke Cwik <lcwik@google.com>
Date: 2017-05-09T02:20:01Z

BEAM-1871 Remove Google API client dependency from sdks/core

This closes #2937

commit 16f355f7e481ebf029c0edb878742f6fea57b6cd
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-08T22:37:29Z

BEAM-2211 DataflowRunner: remove validation of file read/write paths

Now that users can implement and register custom FileSystems,
we can no longer really effectively validate filesystems they
can read or write files from. They can even register file://
to point to some HDFS path, e.g.,

commit 23731fe7ab12ea5d3ebb7b3f16c788b83d0caf0b
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-09T03:13:38Z

This closes #2968

commit 27e9a060ed593b2b53b88481591f14b1a274c61b
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-05-05T23:20:37Z

Remove explicit used of nested contexts.

find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), *context.nested..[)]/\1)/'
find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), *nestedContext[)]/\1)/'
find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), *Context.NESTED[)]/\1)/'
find . -type f -name '.java' | xargs sed -i '' 's/([.]..code[(].), [^ ].Context.NESTED[)]/\1)/'

Added back explicit context in CoGbkResult.java due to compile error.

commit 996dce37b76b103f104328b7caa65f73a1bcb15a
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-05-05T23:36:47Z

Remove contexts from coders where they'll never be used.

commit b7f3341ed4fc073a834a55773bcb4a2c7f821c52
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-05-06T00:24:02Z

automated context removal or redirection

commit 45e09b2df7a97cfbbe0d8013c16cfdd7a55afbde
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-05-06T00:27:13Z

get it compiling

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3172

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3181

BEAM-79 Respect WindowFn#getOutputTime in gearpump-runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam fix_windowing_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3181.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3181

commit 7328ff21f2b805801123c0458d4fad22563ad384
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-05-19T01:19:42Z

BEAM-79 Respect WindowFn#getOutputTime in gearpump-runner

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3292

BEAM-79 Merge master into gearpump-runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3292

commit 66e564e507b1ad5b9e379e043a193b556e1f3e76
Author: Borisa Zivkovic <borisa.zivkovic@huawei.com>
Date: 2017-05-15T08:24:10Z

BEAM-1702 Document how to use BigtableIO with local emulator

commit 8ef812defc5237934f4f3efa6bd8c4c4240cd7d0
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-18T20:07:13Z

This closes #3141

commit a8d2125e5783a556056e88dad8fe3c0a397920d5
Author: huafengw <fvunicorn@gmail.com>
Date: 2017-05-09T03:21:44Z

BEAM-659 WindowFn#isCompatible should provide a meaningful reason

commit 8c572ef0b998a4fa529e2f4ba0e622b3db9f4ec6
Author: Kenneth Knowles <klk@google.com>
Date: 2017-05-18T20:27:59Z

This closes #2985: WindowFn#isCompatible should provide a meaningful reason

commit 26a615e885b39f92dc81d383c8115906c280187a
Author: Andrew Martin <andrewsmartin.mg@gmail.com>
Date: 2017-05-18T19:52:09Z

Remove incorrect `Message` bound on writeAvros

commit 6a7eeeb932ffb33c876c0d11d7f325c47606d8de
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-18T20:52:33Z

This closes #3176: Remove incorrect `Message` bound on writeAvros

commit 790e7fe6653b926044d3dfecdccbc2fda9c998f0
Author: Thomas Groh <tgroh@google.com>
Date: 2017-03-21T22:06:58Z

Add ParDos

Add ParDoPayloadTranslator to PTransformTranslator

commit 2e0cf008303ff5ed48268550b9fe3cd1aa748c15
Author: Thomas Groh <tgroh@google.com>
Date: 2017-05-18T22:41:17Z

This closes #2597

commit 7a2e829c849d3828353994b8eddca4784a900aca
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-15T22:54:03Z

Splits SplittableParDo into a core-construction part and a KWI-related part

commit c93b3aa3d2081558a776fa12c564efb5f15090ab
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-18T23:15:00Z

This closes #3156

commit 54808be14ff2e86088807adc3c0dc0965c8a6f03
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-18T22:43:31Z

BEAM-2318 Ensure that tables that don't exist are not created

We use the same test table name in two tests: one that expects the table not to exist, another that
creates the table. Obviously, the correctness of this will depend on the order in which unit tests
are executed.

Fix the flake by using different and more explicit test names.

commit f458065dad0429ba2639510ea460cf7f2679ed36
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-18T23:41:41Z

This closes #3179

commit 1da8da79dcc623c53b35d97419566b736447f6b6
Author: Dan Halperin <dhalperi@google.com>
Date: 2017-05-18T20:19:29Z

BEAM-2391 Clone Scan in HBaseReader

commit 49245080ad6393b5678077caa047f7a6ef0efffc
Author: Ismaël Mejía <iemejia@apache.org>
Date: 2017-05-19T08:50:06Z

This closes #3178

commit f969ea443c90345cf8c33f953c399e46223e8d03
Author: Luke Cwik <lcwik@google.com>
Date: 2017-05-15T20:39:12Z

BEAM-2252 Use pluginManagement instead of redefining maven-shade-plugin to inherit build order from root pom.xml

commit ea5b9dabbfb55ff47a4f5983a8cbf04b0949c77c
Author: Luke Cwik <lcwik@google.com>
Date: 2017-05-19T17:37:10Z

BEAM-2252 Use pluginManagement instead of redefining maven-shade-plugin to inherit build order from root pom.xml

This closes #3152

commit 3efb34b8322ac5933d9d71255d67e99d8e8361d7
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-19T02:00:00Z

update dataflow python container

commit a58275a2f904ab1304de7ca5295f27a4f6df6f1d
Author: Ahmet Altay <altay@google.com>
Date: 2017-05-19T17:43:15Z

This closes #3182

commit bac7bde5f878e6b2104c2e7cf3d01894a8f1b71d
Author: Mark Liu <markliu@google.com>
Date: 2017-05-17T21:24:25Z

BEAM-2164 Fix generate-sources.sh if directory contains space

commit ce00d2466a9495f4eb262b5f7d2afec5788fcfe2
Author: Kenneth Knowles <klk@google.com>
Date: 2017-05-19T17:44:54Z

This closes #3171: BEAM-2164 Fix generate-sources.sh if directory contains space

commit c7bd2b43ddf726248291a653f8d7d796a51800a0
Author: Kenneth Knowles <klk@google.com>
Date: 2017-05-19T18:40:20Z

Move snappy-java version to dependencyManagement

commit c4d14f8be7bb72fd653f1ab8e8080fc2b65f6672
Author: Colin Phipps <cph@moria.org.uk>
Date: 2017-04-25T15:28:28Z

Comply with byte limit for Datastore Commit.

commit dcf405644e2b51303e1d2c12592fe82ee01eb32f
Author: Colin Phipps <fipsy@google.com>
Date: 2017-05-09T09:40:50Z

End-to-end test for large entity writes.

commit de95c7f5bf3b8391486b898b0ffb1a0a05338725
Author: Colin Phipps <fipsy@google.com>
Date: 2017-05-09T15:07:51Z

Make the large entities test neater.

commit 9a6baefcd8668b145e86140f64b9d18c1c21890c
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-05-19T20:11:50Z

This closes #2948: BEAM-991 Comply with byte limit for Datastore Commit.

commit 383329d6c35e50c8d907ac97522a64236bb98dd0
Author: Luke Cwik <lcwik@google.com>
Date: 2017-05-12T23:44:10Z

BEAM-1345 Remove FileSystems.setDefaultConfigInWorkers since Dataflow no longer depends on this

commit 086148fa9bbdd53fb9684cbbf37346df5a0ec4df
Author: Luke Cwik <lcwik@google.com>
Date: 2017-05-19T20:29:31Z

BEAM-1345 Remove FileSystems.setDefaultConfigInWorkers since Dataflow no longer depends on this

This closes #3134

commit 6c6d7877620af5265ed5ca14dd2ed7cd905b7404
Author: wtanaka.com <wtanaka@yahoo.com>
Date: 2017-05-19T06:42:46Z

BEAM-2293 Rename Top.

This closes #3139

commit 2ed68f7f49a6f84921b0de03a57b0bbe61d263a7
Author: Luke Cwik <lcwik@google.com>
Date: 2017-05-15T22:43:00Z

BEAM-2253 Use resource transformer plugin to use versions defined in root pom.xml within maven archetype poms

Github user manuzhang closed the pull request at:

https://github.com/apache/beam/pull/3181

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3292

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3388

BEAM-79 Sync gearpump-runner branch with master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3388.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3388

commit 8362bdb9cd35cc02ed179b3a64fd72f1264a99be
Author: JingsongLi <lzljs3620320@aliyun.com>
Date: 2017-06-07T17:31:34Z

BEAM-2423 Abstract StateInternalsTest for the different state internals

commit f9d51aa5cc718c44c82ec4773dbd16a57e7d394e
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2017-06-12T20:35:35Z

This closes #3313

commit 809f17876d847002ba76979cb3362451fa01c110
Author: Charles Chen <ccy@google.com>
Date: 2017-06-12T21:17:50Z

Reverse removal of NativeWrite evaluator in Python DirectRunner

commit 86e04893a14d9b6aa29b78e25dc064879b712fb3
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-12T22:43:44Z

This closes #3346

commit f0f98c70b0b01c43ce1b093c2035b20bd90ba907
Author: Thomas Groh <tgroh@google.com>
Date: 2017-06-12T18:07:47Z

Cleanup Combine Tests with Context

Split out the "shared" bit of all the accumulators, so they show up as
an explicit component of the final result string. Update timestamped
creation logic.

commit ce3dd4583294fcbb65121f5fecfc49a8b85634ad
Author: Thomas Groh <tgroh@google.com>
Date: 2017-06-12T22:48:38Z

This closes #3345

commit 1ac18b2eb1371422e60d50a8c3f37b3b24d59611
Author: Thomas Groh <tgroh@google.com>
Date: 2017-06-12T23:55:59Z

Check for Deferral on Non-additional inputs

Because Side Inputs are represented within the expanded inputs, the
check that the transform is a Combine with Side Inputs would never be
hit. This ensures that we do not consider additional inputs during the
check to defer evaluation of the node.

commit 32d55323cd5a88f604ad9c763ab265de845df08c
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-13T03:20:40Z

This closes #3347: Check for Deferral on Non-additional inputs

commit 6859f80400bb16bbca34ed282e7a5e8a1328f955
Author: Ismaël Mejía <iemejia@apache.org>
Date: 2017-06-05T21:20:27Z

BEAM-2412 Update HBaseIO to use HBase client 1.2.6

commit fe3d55403334bf235952fff33e6a90c0f4445a0d
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-06-13T05:59:30Z

BEAM-2412 This closes #3327

commit 4c36508733a69fafce0f7dfb86c71eee5eb6bc84
Author: JingsongLi <lzljs3620320@aliyun.com>
Date: 2017-06-07T06:34:25Z

Use CoderTypeSerializer and remove unuse code in FlinkStateInternals

commit 10b166b355a03daeae78dd1e71016fc72805939d
Author: JingsongLi <lzljs3620320@aliyun.com>
Date: 2017-06-07T06:40:30Z

BEAM-1483 Support SetState in Flink runner and fix MapState to be consistent with InMemoryStateInternals.

commit 4d18606378f43c7b0d3ac05d45ca6e0570e49eef
Author: JingsongLi <lzljs3620320@aliyun.com>
Date: 2017-06-13T02:15:33Z

Add set and map readable test to StateInternalsTest

commit 7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date: 2017-06-13T09:35:43Z

This closes #3309

commit e71eb66ae319bdf0cdad1fe9b54662962c8e8f16
Author: Robert Bradshaw <robertwb@google.com>
Date: 2017-06-09T23:44:55Z

Actually test the fn_api_runner.

The test suite was not being run due to a typo.
Fix breakage due to changes in the code in the meantime.

commit bd83612f246c1654b80df19890a25f8de48a063a
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-06-13T15:56:36Z

Closes #3339

commit b547b5a1eee61e293ff2f8ccfac57f308867328c
Author: Maria Garcia Herrero <mariagh@google.com>
Date: 2017-06-10T06:34:59Z

Make unique test names for value-provider arguments

commit e906fe9c3a1e12f01d84e05fbb872d47b28d2fe9
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-13T16:34:16Z

This closes #3341

commit ee728f1b2f617dac8e5cd729cacf1a46911021e0
Author: Vikas Kedigehalli <vikasrk@google.com>
Date: 2017-06-13T06:11:22Z

Fix WindowValueCoder for large timestamps

commit c33e9b446ef071de02db77883517b308a205c9ff
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-13T17:11:33Z

This closes #3350

commit 33662b92f1a9936f594223c9aee1a7233f59a569
Author: chamikara@google.com <chamikara@google.com>
Date: 2017-06-08T21:56:24Z

Adds ability to dynamically replace PTransforms during runtime.

To this end, adds two interfaces, PTransformMatcher and PTransformOverride.

Currently only supports replacements where input and output types are an exact match (we have to address complexities due to type hints before supporting replacements with different types).

This will be used by SplittableDoFn where matching ParDo transforms will be dynamically replaced by SplittableParDo.

commit 7d0f24a0d91fb1f45027f794d6ba00082f2332ce
Author: chamikara@google.com <chamikara@google.com>
Date: 2017-06-13T18:47:28Z

This closes #3333

commit 3a48e47c73520580021c21a037ab412a761d0eeb
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-06-12T18:51:39Z

Improves message when transitively serializing PipelineOptions

commit 996e35c1de8310a265c890cf8ca8e92416b3d16b
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-06-13T20:04:28Z

This closes #3344: Improves message when transitively serializing PipelineOptions

commit 9115af488ceb907de121313ffa096d58a0ccc1e1
Author: Mairbek Khadikov <mairbek@google.com>
Date: 2017-06-07T23:27:01Z

SpannerIO: Introduced a MutationGroup.

Allows to group together mutation in a logical bundle that is submitted in the same transaction.

commit 646caf255510c735488b88ecc69edd9b9bde4081
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-06-13T20:28:47Z

This closes #3319: BEAM-1542 SpannerIO: Introduced a MutationGroup

commit 7e4e51fda633d208c4cc5e88182e5db16156f2cb
Author: Charles Chen <ccy@google.com>
Date: 2017-06-13T22:03:15Z

Move Runner API protos to portability/api

commit eebff9089e441599b70528b4897c2db1535626c3
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-06-13T23:25:38Z

Closes #3354

commit 8a850af3304a48618739dc23e286800dc0c4641a
Author: Thomas Groh <tgroh@google.com>
Date: 2017-06-13T19:50:58Z

Do not produce Unprocessed Inputs if all inputs were Processed

This stops the WatermarkManager "Pending Bundles" from growing without
bound.

commit dd9abc397c67b47f95414bfd5debb08b7b6a6eaf
Author: Thomas Groh <tgroh@google.com>
Date: 2017-06-14T17:21:02Z

This closes #3353

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3388

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3479

BEAM-79 Merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3479

commit cab4d8969e7f95b0ece59838ad2d578e75d38823
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-22T03:25:31Z

DataflowRunner: Reject merging windowing for stateful ParDo

commit 5f8cfa7412e8f4ce97d6df4413fecbea22b4b5e1
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-23T03:05:28Z

This closes #3419: DataflowRunner: Reject merging windowing for stateful ParDo

commit 649994b353afe28c917969609c7a1a47a4f39aaf
Author: Rune Fevang <fevang@exabel.com>
Date: 2017-06-15T11:51:12Z

Allow output from FinishBundle in DoFnTester

commit b3c36256e32f4a4a616d0e152686d29f1174c5cd
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-23T03:07:48Z

This closes #3367: Allow output from FinishBundle in DoFnTester

commit 8dcda6e40355af13f4d92fcd44aae4539a225a4a
Author: Sourabh Bajaj <sourabhbajaj@google.com>
Date: 2017-06-23T00:08:20Z

BEAM-2497 Fix the reading of concat gzip files

commit 336b7f1cffc8709cd480a243a16de95bce39b53e
Author: Chamikara Jayalath <chamikara@google.com>
Date: 2017-06-23T05:13:05Z

This closes #3428

commit f291713b28e3ba0246a8c0a710c71506cd0a0f91
Author: Etienne Chauchot <echauchot@gmail.com>
Date: 2017-06-21T08:39:39Z

BEAM-2489 Use dynamic ES port in HIFIOWithElasticTest

commit 6cef5c7b625c6b5c2078084ba611b1c2adccc998
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-06-23T13:01:42Z

BEAM-2489 This closes #3411

commit fd8f15f1ac761425dc791a455b042a8846081f48
Author: Mark Liu <markliu@google.com>
Date: 2017-06-22T21:04:00Z

BEAM-2745 Add Jenkins Suite for Python Performance Test

commit af69e979ab3037274bdd78904bdeb5016563bcad
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-23T21:03:17Z

This closes #3426

commit 32095487e56b63b5c1aa690bb6e098375cb108d5
Author: Vikas Kedigehalli <vikasrk@google.com>
Date: 2017-06-23T18:50:12Z

Fix python fn API data plane remote grpc port access

commit 9acce7150ee8fa6c9e50049155ad7b85b646f98e
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-23T22:13:30Z

This closes #3431

commit 903da41ac5395e76c44ef8ae1c8a695569e23abb
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-06-23T22:01:42Z

Avoid pickling the entire pipeline per-transform.

commit a90e40ae9f748c1c8392c1198469f3229a06ed70
Author: Robert Bradshaw <robertwb@gmail.com>
Date: 2017-06-23T23:39:52Z

Closes #3433

commit e45f522d6e945899c20259ebf8faca105c2e552e
Author: Valentyn Tymofieiev <valentyn@google.com>
Date: 2017-06-23T23:44:49Z

Fix a typo in function args

commit 16f87f49f20796e29d01ed363a9097ea5420583c
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-24T00:34:21Z

This closes #3435

commit 926f949580c3a21df72a8836feda1f6b947850ec
Author: Charles Chen <ccy@google.com>
Date: 2017-06-26T20:00:14Z

Remove old deprecated PubSub code

commit 1ea1de4aa9d32e3c5a596ccd7d84aff1cc2a7428
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-26T21:23:08Z

This closes #3440

commit bec32fe93c6b5c16563d7ea4b877a2dee3352fee
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-06-16T21:56:07Z

Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part)

commit 2052cc7689b4ed53f817f56dd71a32235fb083ca
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-06-23T17:16:30Z

Bump Dataflow worker to 0623

commit 95e6bbe5065a983f9c71b79b20df1fda83c9fd1b
Author: Eugene Kirpichov <kirpichov@google.com>
Date: 2017-06-27T00:25:14Z

This closes #3383: BEAM-2447 Reintroduces DoFn.ProcessContinuation (Dataflow worker compatibility part)

commit eb379e76adaa9c4b4e24a4b3c5757be8523d95c4
Author: Charles Chen <ccy@google.com>
Date: 2017-06-26T23:54:00Z

Implement streaming GroupByKey in Python DirectRunner

commit 8036001da6f90eac20d787a05d35a51e30146278
Author: Ahmet Altay <altay@google.com>
Date: 2017-06-27T05:49:19Z

This closes #3444

commit cbfcad823972270171552e556b7fa8d8d4882f14
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2016-12-28T14:06:09Z

BEAM-1237 Create AmqpIO

commit 9df865add9bdd83d265d0f8442e6b07467bcf17d
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-06-27T13:33:51Z

BEAM-1237 This closes #1725

commit 01b3febc62debf8b4fd1582225d59768d231601a
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-06-26T12:55:13Z

BEAM-2514 Improve error message on missing required value

commit 99ef92bc97692bdba3ea8e10cc09d81f3d64b69b
Author: Luke Cwik <lcwik@google.com>
Date: 2017-06-27T16:23:03Z

BEAM-2514 Improve error message on missing required value

This closes #3438

commit d33d64644c8ddc74a3dc07bf0c642a9ed818a624
Author: Nigel Kilmer <nkilmer@google.com>
Date: 2017-06-21T18:26:10Z

Removed uses of proto builder clone method

commit e93c06485508ae3a7a8304be3d6127d604d9356e
Author: Luke Cwik <lcwik@google.com>
Date: 2017-06-27T16:40:53Z

BEAM-2392 Remove uses of proto builder clone

This closes #3415

commit 8dd0077d2a58e278b11c7e7eb4b5f182e1400992
Author: Vikas Kedigehalli <vikasrk@google.com>
Date: 2017-06-27T01:47:39Z

Enable grpc controller in fn_api_runner

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3479

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3515

BEAM-79 Sync gearpump-runner with master and upgrade to gearpump 0.8.4

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• [ ] Make sure the PR title is formatted like:
  `BEAM-<Jira issue #> Description of pull request`
• [ ] Make sure tests pass via `mvn clean verify`.
• [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
  number, if there is one.
• [ ] If this contribution is large, please file an Apache
  [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3515

commit 7b8cd6401cb5ed6e184ed36571a89d3ae324dd5f
Author: Jeremie Lenfant-Engelmann <jeremiele@google.com>
Date: 2017-06-29T01:32:56Z

Properly convert milliseconds whether there's less than 3/more than 9 digits.
TimeUtil did not properly convert (and returned null) when the number of
digits for fractions of seconds was less than 3 digits or more than 9 digits.
The solution is to pad with zeros when there is less than 3 digits and to
truncate when there is more than 3.

commit f75dfe711b357800762a93dc5059d9a19da9ee61
Author: Luke Cwik <lcwik@google.com>
Date: 2017-06-30T16:00:13Z

Properly convert milliseconds whether there's less than 3/more than 9 digits.

This closes #3469

commit 56cb6c51748fde6ad56522733ab10edca062e802
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-13T17:29:50Z

Add support for PipelineOptions parameters

This is a step towards eliminating catch-all context parameters and
making DoFns express their fine-grained data needs.

commit 0e429b33ff85eba08da5018c9febd0b99b44f720
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-30T22:39:20Z

This closes #3355: Add support for PipelineOptions parameters

commit f99ab1a472868e4ce175a86e5d76823b1c09c10b
Author: Kenneth Knowles <klk@google.com>
Date: 2017-07-01T04:42:17Z

Fix DoFn javadoc: StateSpec does not require a key

commit ce4e51747501111ae2c4b1691c6994bd0f92e161
Author: Ismaël Mejía <iemejia@apache.org>
Date: 2017-06-04T20:55:05Z

Make modules that depend on Hadoop and Spark use the same version property

commit 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-07-03T07:26:40Z

This closes #3291

commit 75475ef3dc23a09fa9bbba478d6fdbc468f7dd2e
Author: Ismaël Mejía <iemejia@gmail.com>
Date: 2017-06-28T14:58:55Z

BEAM-2530 Fix compilation of modules with Java 9 that depend on jdk.tools

commit 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-07-03T12:23:18Z

BEAM-2530 This closes #3460

commit 68f1fb64fd2565e287e322d715ca778d01e7137b
Author: Ahmet Altay <altay@google.com>
Date: 2017-07-01T00:37:33Z

For GCS operations use an http client with a default timeout value.

commit b8ac32641183b06a87517a169c8ecc5371996aed
Author: Ahmet Altay <altay@google.com>
Date: 2017-07-04T18:07:31Z

This closes #3486

commit 51877a3405dbf778c3bb88f19bb194e54c3b3def
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-07-05T14:47:29Z

[maven-release-plugin] prepare branch release-2.1.0

commit 7f0723cf7fd587c4f8dbe8a8b1c9d298a7b1e5e3
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-07-05T14:47:38Z

[maven-release-plugin] prepare for next development iteration

commit 14fa7f79f0830739122b7573e032ad0aea172a98
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Date: 2017-07-05T14:52:48Z

Update Python SDK version

commit 06897b1cc142f658437ac7779c849e5182e331f1
Author: Jason Kuster <jasonkuster@google.com>
Date: 2017-06-09T08:39:15Z

Website Mergebot Job

Signed-off-by: Jason Kuster <jasonkuster@google.com>

commit f0549b464737d0d11ae1da8e0534622b10247208
Author: Thomas Groh <tgroh@google.com>
Date: 2017-07-05T16:22:20Z

This closes #3485

commit 6ca410a908f1f4e7ac1e141ee1335f7a537bb150
Author: Luke Cwik <lcwik@google.com>
Date: 2017-07-05T17:38:44Z

BEAM-2553 Update Maven exec plugin to 1.6.0 to incorporate messaging improvements

commit 39c80f640749fed5f55a0c8f25c849e5f7c3d179
Author: Luke Cwik <lcwik@google.com>
Date: 2017-07-05T19:26:28Z

BEAM-2553 Update Maven exec plugin to 1.6.0 to incorporate messaging improvements

This closes #3494

commit 29c2bca4649317f2ebb1c89f92bf97fbb27602ca
Author: Thomas Groh <tgroh@google.com>
Date: 2017-07-05T21:16:50Z

Disallow Combiner Lifting for multi-window WindowFns

commit 63f66d6d81652dba5a7804648cde0e7debffd0f6
Author: Kenneth Knowles <klk@google.com>
Date: 2017-07-05T22:00:09Z

This closes #3489: Fix DoFn javadoc: StateSpec does not require a key

commit 23e385faa193a00f9b10e3f8f0afe832087bff06
Author: Ahmet Altay <altay@google.com>
Date: 2017-07-05T21:34:07Z

Update SDK dependencies

commit ea9f7fbdde5ede5e5f5060da5dfe2749992a2fa7
Author: Ahmet Altay <altay@google.com>
Date: 2017-07-05T23:11:46Z

This closes #3497

commit e8fdd373c7b86652fda94ff76ef7246357e6288c
Author: Kenneth Knowles <klk@google.com>
Date: 2017-07-05T23:41:16Z

This closes #3496: Disallow Combiner Lifting for multi-window WindowFns

commit a75202f344f22be5c5fdf62b3eb54a151ad29af6
Author: Charles Chen <ccy@google.com>
Date: 2017-07-05T23:18:51Z

Fix PValue input in _PubSubReadEvaluator

commit bf6dda32047e0bfba18de5aa27b1b6a593f86fc9
Author: Ahmet Altay <altay@google.com>
Date: 2017-07-06T01:09:05Z

This closes #3499

commit 2259c309c5b81a5d1e32732dd35e1102766401fa
Author: Raghu Angadi <rangadi@google.com>
Date: 2017-06-28T19:07:06Z

BEAM-2534 Handle offset gaps in Kafka messages.

KafkaIO logged a warning when there is a gap in offstes for messages.
Kafka also support 'KV' store style topics where some of the messages
are deleted leading gaps in offsets. This PR removes the log and
accounts for offset gaps in backlog estimate.

commit 85a99e29448670ae6728a8ee2e4cd3ef95877c3e
Author: JingsongLi <lzljs3620320@aliyun.com>
Date: 2017-07-06T16:48:36Z

This closes #3461

commit 526037b6786315b9f9fdca6edb636baeb6f83e3f
Author: Raghu Angadi <rangadi@google.com>
Date: 2017-07-04T06:54:10Z

Add timeout to initialization of partition in KafkaIO

commit b8f8d18ae2cdbb4874d9a0f45038037ecc2381d1
Author: Kenneth Knowles <klk@google.com>
Date: 2017-07-06T18:59:18Z

This closes #3492: BEAM-2551 KafkaIO reader blocks indefinitely in case of network issues

Add timeout to initialization of partition in KafkaIO

commit bd631b89a8434f0756e1596875e89013fb623ab5
Author: Kenneth Knowles <klk@google.com>
Date: 2017-06-23T01:09:11Z

Ignore processing time timers in expired windows

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3515

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3611

BEAM-79 merge gearpump-runner into master

Follow this checklist to help us incorporate your contribution quickly and easily:

• [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
• [ ] Each commit in the pull request should have a meaningful subject line and body.
• [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
• [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam gearpump-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3611.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3611

commit 9478f4117de3a2d0ea40614ed4cb801918610724
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-03-15T08:15:16Z

BEAM-79 add Gearpump runner

commit 02b2248a5b3c8a2c064547d7380bebc97f849bf1
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-20T16:07:48Z

This closes #323

commit 2a0ba61e8507e1539115b583749a78f14d577bd8
Author: Kenneth Knowles <klk@google.com>
Date: 2016-08-25T18:36:45Z

Merge branch master into gearpump-runner

commit 1672b5483e029292816397248dc6fe63bf51f4af
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-07-23T06:10:15Z

move integration tests to profile

commit 276a2e106aa1a5736666fc2eb2426b640f63cf68
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-07-28T08:30:13Z

add package-info.java

commit 40be715a696bb1218b209f7ad9a979b7e5d088d3
Author: Kenneth Knowles <klk@google.com>
Date: 2016-08-10T17:26:57Z

Update Gearpump runner version to 0.3.0-incubating

commit bc1b354949416db3b52c4f37c66968bdb86f0813
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-08-11T23:22:00Z

Rename DoFn to OldDoFn in Gearpump runner

commit 091a15a07c7625ae3009cefaecece3a29a34c109
Author: Kenneth Knowles <klk@google.com>
Date: 2016-08-25T18:40:03Z

This closess #750

commit fb74c936ed92c7a8548c338cc03957794fc60902
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-08-26T23:25:58Z

gearpump: switch to stable version

They have apparently deleted the SNAPSHOT jar and now builds are failing.

commit bf0a2edae11416a3cbddeaff2c0a70adc272c5fe
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-08-27T00:46:42Z

Closes #895

commit 89921c41ca9d4c333af45efa32359a631214c1df
Author: bchambers <bchambers@google.com>
Date: 2016-07-29T16:41:17Z

Remove Counter and associated code

Aggregator is the model level concept. Counter was specific to the
Dataflow Runner, and is now not needed as part of Beam.

commit 7fc2c6848f002ac8b2ccbe35e6b5a576777a7af9
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-03T00:25:14Z

BEAM-495 Create General Verifier for File Checksum

commit b47549e4893a6d487c00ea0ba02619168a3f19f3
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-03T00:47:46Z

Add output checksum to WordCountITOptions

commit 58cd781c82fa728f34f5ab0641f8f9b6edcf449c
Author: Ian Zhou <ianzhou@google.com>
Date: 2016-08-05T22:31:59Z

Added unit tests and error handling in removeTemporaryTables

commit 36a9aa232ea56de449930194788becce585212ef
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-09T02:09:58Z

Improve Write Error Message

If provided with an Unbounded PCollection, Write will fail due to
restriction of calling finalize only once. This error message fails in a
deep stack trace based on it not being possible to apply a GroupByKey.
Instead, throw immediately on application with a specific error message.

commit d5641553cebb02f08ca7c1fe667948d39cb3962c
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-09T17:47:09Z

Remove Streaming Write Overrides in DataflowRunner

These writes should be forbidden based on the boundedness of the input
PCollection. As Write explicitly forbids the application of the
transform to an Unbounded PCollection, this will be equivalent in most
cases; In cases where the input PCollection is Bounded, due to an
UnboundedReadFromBoundedSource, the write will function as expected and
does not need to be forbidden.

commit 011bea9a83a828e0d8c6518ab83aa5cc4f75e146
Author: David Rieber <drieber@google.com>
Date: 2016-08-09T21:05:25Z

Do not add DataDisks to windmill service jobs.

commit 0dfb8ff55d6f80264222fde4501ea3050d2e3911
Author: gaurav gupta <gaugupt3@cisco.com>
Date: 2016-08-10T23:43:03Z

Made byteArrayCoder final static

commit b9f826366823003940805e6469b10df8819b0977
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-08-11T00:58:09Z

CompressedSource: CompressedReader is never splittable

The only way it's safe to split a compressed file is if the file is not compressed. This can
only happen when the source itself is splittable, and that in turn will result in the inner
source's reader being returned. A CompressedReader will only be created in the event that
the file is NOT splittable. So remove all the logic handling splittable compressed readers,
and instead go with the logic when we know/assume the file is compressed.

• TextIO: test compression with larger files

It is important for correctness that we test with large files
because otherwise the compressed file may be larger than the
uncompressed file, which could mask bugs

• TextIOTest: flesh out more

• TextIOTest: add large uncompressed file

commit 1d86335314685926d3eb0f9765d615a77cee75e6
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-11T16:16:55Z

Remove timeout in DirectRunnerTest

If the test hangs due to bugs, the infrastructure should kill it.

commit 37ce2a3e75cb96a4b3fdcd4938fb7fda95122724
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-11T18:26:28Z

More unit test and code style fix

commit 046e36eaa41659ae43866bc6fbce4f122889f286
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-11T18:55:17Z

Using IOChannelUtils to resolve file path

commit d056f4661da2cc399cab44c6604eaa61d1dfd178
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-14T21:51:02Z

Add DoFn @Setup and @Teardown

Methods annotated with these annotations are used to perform expensive
setup work and clean up a DoFn after another method throws an exception
or the DoFn is discarded.

commit b80d96748dcb71f93697126489c924020ebbd4a9
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-15T18:27:00Z

Add TransformEvaluatorFactory#cleanup

This cleans up any state stored within the Transform Evaluator Factory.

commit 77c90d00ae715795e73efec8f8e85e3917cf8d80
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-19T18:03:15Z

Replace CloningThreadLocal with DoFnLifecycleManager

This is a more focused interface that interacts with a DoFn before it
is available for use and after it has completed and the reference is
lost. It is required to properly support setup and teardown, as the
fields in a ThreadLocal cannot all be cleaned up without additional
tracking.

Part of BEAM-452.

commit 39f763e16182e33019a1805d6210549934998856
Author: Pei He <peihe@google.com>
Date: 2016-08-01T20:41:59Z

Remove DataflowPipelineJob from examples

commit 6603307062ec99639d3e3e05aebc0d1ea32ad411
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-11T17:45:43Z

Move ParDo Lifecycle tests to their own file

These tests are not yet functional in all runners, and this makes them
easier to ignore.

commit d99a652f96f12bcc235caa038ffa741906336b1f
Author: Maximilian Michels <mxm@apache.org>
Date: 2016-08-12T15:51:02Z

[flink] add missing maven config to example pom

commit 424c4c492965e5a93b1c020c8d52805e3a9a9088
Author: mariusz89016 <mariusz89016@gmail.com>
Date: 2016-08-13T22:35:19Z

BEAM-432 Corrected BigQueryIO javadoc

commit d6cf4f2a3cda9ff4ff2da105c08be8101c58e6f1
Author: Daniel Halperin <dhalperi@users.noreply.github.com>
Date: 2016-08-15T22:21:15Z

Exclude ParDoTest from Dataflow @RunnableOnService

Until we implement it for Dataflow runner.

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3636

BEAM-79 merge gearpump-runner into master

Follow this checklist to help us incorporate your contribution quickly and easily:

• [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
• [ ] Each commit in the pull request should have a meaningful subject line and body.
• [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
• [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/beam gearpump-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3636

commit 9478f4117de3a2d0ea40614ed4cb801918610724
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-03-15T08:15:16Z

BEAM-79 add Gearpump runner

commit 02b2248a5b3c8a2c064547d7380bebc97f849bf1
Author: Kenneth Knowles <klk@google.com>
Date: 2016-07-20T16:07:48Z

This closes #323

commit 2a0ba61e8507e1539115b583749a78f14d577bd8
Author: Kenneth Knowles <klk@google.com>
Date: 2016-08-25T18:36:45Z

Merge branch master into gearpump-runner

commit 1672b5483e029292816397248dc6fe63bf51f4af
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-07-23T06:10:15Z

move integration tests to profile

commit 276a2e106aa1a5736666fc2eb2426b640f63cf68
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-07-28T08:30:13Z

add package-info.java

commit 40be715a696bb1218b209f7ad9a979b7e5d088d3
Author: Kenneth Knowles <klk@google.com>
Date: 2016-08-10T17:26:57Z

Update Gearpump runner version to 0.3.0-incubating

commit bc1b354949416db3b52c4f37c66968bdb86f0813
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2016-08-11T23:22:00Z

Rename DoFn to OldDoFn in Gearpump runner

commit 091a15a07c7625ae3009cefaecece3a29a34c109
Author: Kenneth Knowles <klk@google.com>
Date: 2016-08-25T18:40:03Z

This closess #750

commit fb74c936ed92c7a8548c338cc03957794fc60902
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-08-26T23:25:58Z

gearpump: switch to stable version

They have apparently deleted the SNAPSHOT jar and now builds are failing.

commit bf0a2edae11416a3cbddeaff2c0a70adc272c5fe
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-08-27T00:46:42Z

Closes #895

commit 89921c41ca9d4c333af45efa32359a631214c1df
Author: bchambers <bchambers@google.com>
Date: 2016-07-29T16:41:17Z

Remove Counter and associated code

Aggregator is the model level concept. Counter was specific to the
Dataflow Runner, and is now not needed as part of Beam.

commit 7fc2c6848f002ac8b2ccbe35e6b5a576777a7af9
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-03T00:25:14Z

BEAM-495 Create General Verifier for File Checksum

commit b47549e4893a6d487c00ea0ba02619168a3f19f3
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-03T00:47:46Z

Add output checksum to WordCountITOptions

commit 58cd781c82fa728f34f5ab0641f8f9b6edcf449c
Author: Ian Zhou <ianzhou@google.com>
Date: 2016-08-05T22:31:59Z

Added unit tests and error handling in removeTemporaryTables

commit 36a9aa232ea56de449930194788becce585212ef
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-09T02:09:58Z

Improve Write Error Message

If provided with an Unbounded PCollection, Write will fail due to
restriction of calling finalize only once. This error message fails in a
deep stack trace based on it not being possible to apply a GroupByKey.
Instead, throw immediately on application with a specific error message.

commit d5641553cebb02f08ca7c1fe667948d39cb3962c
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-09T17:47:09Z

Remove Streaming Write Overrides in DataflowRunner

These writes should be forbidden based on the boundedness of the input
PCollection. As Write explicitly forbids the application of the
transform to an Unbounded PCollection, this will be equivalent in most
cases; In cases where the input PCollection is Bounded, due to an
UnboundedReadFromBoundedSource, the write will function as expected and
does not need to be forbidden.

commit 011bea9a83a828e0d8c6518ab83aa5cc4f75e146
Author: David Rieber <drieber@google.com>
Date: 2016-08-09T21:05:25Z

Do not add DataDisks to windmill service jobs.

commit 0dfb8ff55d6f80264222fde4501ea3050d2e3911
Author: gaurav gupta <gaugupt3@cisco.com>
Date: 2016-08-10T23:43:03Z

Made byteArrayCoder final static

commit b9f826366823003940805e6469b10df8819b0977
Author: Dan Halperin <dhalperi@google.com>
Date: 2016-08-11T00:58:09Z

CompressedSource: CompressedReader is never splittable

The only way it's safe to split a compressed file is if the file is not compressed. This can
only happen when the source itself is splittable, and that in turn will result in the inner
source's reader being returned. A CompressedReader will only be created in the event that
the file is NOT splittable. So remove all the logic handling splittable compressed readers,
and instead go with the logic when we know/assume the file is compressed.

• TextIO: test compression with larger files

It is important for correctness that we test with large files
because otherwise the compressed file may be larger than the
uncompressed file, which could mask bugs

• TextIOTest: flesh out more

• TextIOTest: add large uncompressed file

commit 1d86335314685926d3eb0f9765d615a77cee75e6
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-11T16:16:55Z

Remove timeout in DirectRunnerTest

If the test hangs due to bugs, the infrastructure should kill it.

commit 37ce2a3e75cb96a4b3fdcd4938fb7fda95122724
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-11T18:26:28Z

More unit test and code style fix

commit 046e36eaa41659ae43866bc6fbce4f122889f286
Author: Mark Liu <markliu@markliu-macbookpro.roam.corp.google.com>
Date: 2016-08-11T18:55:17Z

Using IOChannelUtils to resolve file path

commit d056f4661da2cc399cab44c6604eaa61d1dfd178
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-14T21:51:02Z

Add DoFn @Setup and @Teardown

Methods annotated with these annotations are used to perform expensive
setup work and clean up a DoFn after another method throws an exception
or the DoFn is discarded.

commit b80d96748dcb71f93697126489c924020ebbd4a9
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-15T18:27:00Z

Add TransformEvaluatorFactory#cleanup

This cleans up any state stored within the Transform Evaluator Factory.

commit 77c90d00ae715795e73efec8f8e85e3917cf8d80
Author: Thomas Groh <tgroh@google.com>
Date: 2016-07-19T18:03:15Z

Replace CloningThreadLocal with DoFnLifecycleManager

This is a more focused interface that interacts with a DoFn before it
is available for use and after it has completed and the reference is
lost. It is required to properly support setup and teardown, as the
fields in a ThreadLocal cannot all be cleaned up without additional
tracking.

Part of BEAM-452.

commit 39f763e16182e33019a1805d6210549934998856
Author: Pei He <peihe@google.com>
Date: 2016-08-01T20:41:59Z

Remove DataflowPipelineJob from examples

commit 6603307062ec99639d3e3e05aebc0d1ea32ad411
Author: Thomas Groh <tgroh@google.com>
Date: 2016-08-11T17:45:43Z

Move ParDo Lifecycle tests to their own file

These tests are not yet functional in all runners, and this makes them
easier to ignore.

commit d99a652f96f12bcc235caa038ffa741906336b1f
Author: Maximilian Michels <mxm@apache.org>
Date: 2016-08-12T15:51:02Z

[flink] add missing maven config to example pom

commit 424c4c492965e5a93b1c020c8d52805e3a9a9088
Author: mariusz89016 <mariusz89016@gmail.com>
Date: 2016-08-13T22:35:19Z

BEAM-432 Corrected BigQueryIO javadoc

commit d6cf4f2a3cda9ff4ff2da105c08be8101c58e6f1
Author: Daniel Halperin <dhalperi@users.noreply.github.com>
Date: 2016-08-15T22:21:15Z

Exclude ParDoTest from Dataflow @RunnableOnService

Until we implement it for Dataflow runner.

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/3637

BEAM-79 Prepare for merging gearpump-runner

Follow this checklist to help us incorporate your contribution quickly and easily:

• [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
• [ ] Each commit in the pull request should have a meaningful subject line and body.
• [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
• [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/beam gearpump-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3637

commit e655f53c2e732bcd082449ba8d0b551f417e8aaa
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-07-21T06:27:37Z

Revert accidental changes to sdks/java/pom.xml

commit daa7566939e212095ef745ff004580bfe4209b38
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-07-21T06:58:13Z

Upgrade BEAM version to 2.2.0-SNAPSHOT in gearpump-runner

commit 49d4ed59567852e978d14729c89b9ea91bb96c32
Author: manuzhang <owenzhang1990@gmail.com>
Date: 2017-07-21T14:33:36Z

Add beam-runners-gearpump dependency to javadoc

Github user manuzhang closed the pull request at:

https://github.com/apache/beam/pull/3611

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3637

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3636

Hi Tyler Akidau,

We've reached a milestone after merging gearpump-runner into master. Future development will be under new JIRAs. Can we close this issue now ?

Thanks for asking on Twitter and initiating this work a year ago.

Manu

GitHub user huafengw opened a pull request:

https://github.com/apache/beam/pull/3726

BEAM-79 Add Gearpump runner profile to Java8 Archetype POM

Follow this checklist to help us incorporate your contribution quickly and easily:

• [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
• [ ] Each commit in the pull request should have a meaningful subject line and body.
• [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
• [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

—

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huafengw/beam example

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3726.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3726

commit 31810c459ea95cb7816261087a7e73dc0e0338c3
Author: huafengw <fvunicorn@gmail.com>
Date: 2017-08-17T03:58:33Z

BEAM-79 Add Gearpump runner profile to Java8 Archetype POM

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3726