[BEAM-160] Port 'NexMark Queries' to Beam for use as integration test

• Type: Test
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: testing, testing-nexmark
• Assignee: Etienne Chauchot
• Reporter: Mark Shields
• Created: 30/Mar/16 19:54
• Updated: 16/May/20 13:46
• Resolved: 24/Aug/17 09:19

Description

A while back we implemented the 'queries' from
http://datalab.cs.pdx.edu/niagara/NEXMark/
as Gooogle Dataflow pipelines. We found them useful
for uncovering performance problems with the sdk, our runners,
and our service. Many of those problems only manifested under
high load, multi-day runs, or with high 'backlog' on the incoming
pub/sub subscriptions.

We thus think they would be useful for other runners.

Disclaimer: Though the original 'queries' were proposed as a way to
benchmark 'continuous SQL' implementations, we have so far only
used them for internal A/B and regression testing and have not validated
them as representative of customer workloads. We would thus discourage their use for competitive benchmarks without more work.

Issue Links

• relates to: BEAM-775 Remove Aggregators from the Java SDK
• relates to: BEAM-2111 ClassCastException in streaming Spark pipeline.
• relates to: BEAM-2499 Support Custom Windows in Spark runner
• relates to: BEAM-2408 Flink unbounded source does not emit watermarks when there are multiple Readers
• relates to: BEAM-1772 Support merging WindowFn other than IntervalWindow on Flink Runner
• relates to: BEAM-1802 Spark Runner does not shutdown correctly when executing multiple pipelines in sequence
• relates to: BEAM-1810 Spark runner combineGlobally uses Kryo serialization
• relates to: BEAM-2029 NullPointerException when using multi output ParDo in Spark runner in streaming mode.
• relates to: BEAM-2409 Spark runner produces exactly twice the number of results in streaming mode when use triggers to re-window results on global window.
• relates to: BEAM-1948 Null pointer exception in DirectRunner.DirectPipelineResult.getAggregatorValues()
• relates to: BEAM-593 Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult
• relates to: BEAM-1035 Support for new State API in SparkRunner
• relates to: BEAM-1115 Support for new Timer API in Spark runner
• relates to: BEAM-1037 Support for new State API in ApexRunner
• relates to: BEAM-1114 Support for new Timer API in Apex runner
• relates to: BEAM-2112 Add support for PCollectionView in spark runner in streaming mode
• relates to: BEAM-2847 Stop streaming pipeline when global watermark reaches +INF
• relates to: BEAM-2971 Nexmark: migrate WinningBids to state API in place of custom window merge.
• relates to: BEAM-2972 Nexmark: create a query that illustrates a simple custom window merge
• relates to: BEAM-829 Implement Metrics support for Apex runner
• relates to: BEAM-1261 State API should allow state to be managed in different windows
• links to: GitHub Pull Request #366
• links to: GitHub Pull Request #3114