[SPARK-8360] Structured Streaming (aka Streaming DataFrames)

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 2.1.0
• Component/s: Structured Streaming
• Assignee: Michael Armbrust
• Reporter: Reynold Xin
• Created: 14/Jun/15 07:26
• Updated: 01/Nov/16 23:44
• Resolved: 01/Nov/16 23:44

Description

Umbrella ticket to track what's needed to make streaming DataFrame a reality.

Issue Links

• incorporates: SPARK-16350 Complete output mode does not output updated aggregated value in Structured Streaming
• is duplicated by: SPARK-1363 Add streaming support for Spark SQL module
• relates to: SPARK-9999 Dataset API on top of Catalyst/DataFrame
• links to: Structured Streaming Programming Abstraction, Semantics, and APIs - Google Docs version

Sub-Tasks

• API design: convergence of batch and streaming DataFrame: Resolved
• Initial infrastructure: Resolved
• API design: external state management: Closed
• API for managing streaming dataframes: Resolved
• Add FileStreamSource: Resolved
• Remove DataStreamReader/Writer: Resolved
• Rename DataFrameWriter.stream DataFrameWriter.startStream: Resolved
• State Store: A new framework for state management for computing Streaming Aggregates: Resolved
• Old streaming DataFrame proposal by Cheng Hao (Intel): Closed
• WAL for determistic batches with IDs: Resolved
• Simple FileSink for Parquet: Resolved
• Windowing for structured streaming: Resolved
• Add processing time trigger: Resolved
• Streaming Aggregation: Resolved
• Method to determine if Dataset is bounded or not: Resolved
• Memory Sink: Resolved
• Define analysis rules for operations not supported in streaming: Resolved
• Python API for methods introduced for Structured Streaming: Resolved
• Add partitioned parquet support file stream sink: Resolved
• Refactor DataSource to ensure schema is inferred only once when creating a file stream: Resolved
• Refactor StreamTests to test for source fault-tolerance correctly.: Resolved
• Add support in file stream source for reading new files added to subdirs: Resolved
• Add support for batch jobs correctly inferring partitions from data written with file stream sink: Resolved
• Disable support for multiple streaming aggregations: Resolved
• Disable schema inference for streaming datasets on file streams: Resolved
• Add support for complete output mode: Resolved
• Make continuous Parquet writes consistent with non-continuous Parquet writes: Closed
• Allow sorting on aggregated streaming dataframe when the output mode is Complete: Resolved
• Add support for socket stream.: Closed
• Add DataFrameWriter.foreach to allow the user consuming data in ContinuousQuery: Resolved
• Add a unique id to ContinuousQuery: Resolved
• Refactor reader-writer interface for streaming DFs to use DataStreamReader/Writer: Resolved
• Renamed ContinuousQuery to StreamingQuery for simplicity: Resolved
• Fix bug in python DataStreamReader: Resolved
• Properly explain the streaming queries: Resolved
• Fix complete mode aggregation with console sink: Resolved
• Sleep when no new data arrives to avoid 100% CPU usage: Resolved
• Enable test for sql/streaming.py and fix these tests: Resolved
• HDFSMetadataLog.get leaks the input stream: Resolved
• Add ContinuousQueryInfo to make ContinuousQueryListener events serializable: Resolved
• Add network word count example: Resolved
• StreamExecution.awaitOffset may take too long because of thread starvation: Resolved
• Fix flaky test: o.a.s.sql.util.ContinuousQueryListenerSuite "event ordering": Resolved
• Add a file sink log to support versioning and compaction: Resolved
• Fix a race condition in StreamExecution.processAllAvailable: Resolved
• Fix the race conditions in MemoryStream and MemorySink: Resolved
• Move FileSource offset log into checkpointLocation: Resolved
• Add a note to warn that onQueryProgress is asynchronous: Resolved
• QueryProgress should be post after committedOffsets is updated: Resolved
• StateStoreCoordinator should extend ThreadSafeRpcEndpoint: Resolved
• Allow multiple continuous queries to be started from the same DataFrame: Resolved
• Add a workaround for HADOOP-10622 to fix DataFrameReaderWriterSuite: Resolved
• Add MetadataLog and HDFSMetadataLog: Resolved
• ContinuousQueryManagerSuite floods the logs with garbage: Resolved
• Flaky test: o.a.s.sql.util.ContinuousQueryListenerSuite.event ordering: Resolved
• Add ConsoleSink for structure streaming to display the dataframe on the fly: Resolved
• Flaky Test: Complete aggregation with Console sink: Resolved
• ConsoleSink should not require checkpointLocation: Resolved
• Add Structured Streaming Programming Guide: Resolved
• Move python DataStreamReader/Writer from pyspark.sql to pyspark.sql.streaming package: Resolved
• Add an option in file stream source to read 1 file at a time: Resolved
• Fix StreamingQueryListener to return message and stacktrace of actual exception: Resolved
• Running a file stream on a directory with partitioned subdirs throw NotSerializableException/StackOverflowError: Resolved
• Metrics for Structured Streaming: Resolved
• Add methods to convert StreamingQueryStatus to json: Resolved
• History Server is broken because of the refactoring work in Structured Streaming: Resolved
• ForeachSink should fail the Spark job if `process` throws exception: Resolved
• State Store leaks temporary files: Resolved
• Fix FileStreamSink with aggregation + watermark + append mode: Resolved
• Rename triggerId to batchId in StreamingQueryStatus.triggerDetails: Resolved
• Include triggerDetails in StreamingQueryStatus.json: Resolved
• Improve docs on StreamingQueryListener and StreamingQuery.status: Resolved
• Add StreamingQuery.status in python: Closed
• Enable interrupts for HDFS in HDFSMetadataLog: Resolved