Bulk Operation

SPARK-8360 Enable interrupts for HDFS in HDFSMetadataLog

SPARK-8360 Add StreamingQuery.status in python

SPARK-8360 Improve docs on StreamingQueryListener and StreamingQuery.status

SPARK-8360 Include triggerDetails in StreamingQueryStatus.json

SPARK-8360 Rename triggerId to batchId in StreamingQueryStatus.triggerDetails

SPARK-8360 Fix FileStreamSink with aggregation + watermark + append mode

SPARK-8360 State Store leaks temporary files

SPARK-8360 ForeachSink should fail the Spark job if `process` throws exception

SPARK-8360 History Server is broken because of the refactoring work in Structured Streaming

SPARK-8360 Add methods to convert StreamingQueryStatus to json

SPARK-8360 Metrics for Structured Streaming

SPARK-8360 Running a file stream on a directory with partitioned subdirs throw NotSerializableException/StackOverflowError

SPARK-8360 Fix StreamingQueryListener to return message and stacktrace of actual exception

SPARK-8360 Add an option in file stream source to read 1 file at a time

SPARK-8360 Move python DataStreamReader/Writer from pyspark.sql to pyspark.sql.streaming package

SPARK-8360 Add Structured Streaming Programming Guide

SPARK-8360 ConsoleSink should not require checkpointLocation

SPARK-8360 Add network word count example

SPARK-8360 Flaky Test: Complete aggregation with Console sink

SPARK-8360 Fix complete mode aggregation with console sink

SPARK-8360 Sleep when no new data arrives to avoid 100% CPU usage

SPARK-8360 Fix bug in python DataStreamReader

SPARK-8360 Renamed ContinuousQuery to StreamingQuery for simplicity

SPARK-8360 Enable test for sql/streaming.py and fix these tests

SPARK-8360 Refactor reader-writer interface for streaming DFs to use DataStreamReader/Writer

SPARK-8360 Add a unique id to ContinuousQuery

SPARK-8360 HDFSMetadataLog.get leaks the input stream

SPARK-8360 Add support for socket stream.

SPARK-8360 Allow sorting on aggregated streaming dataframe when the output mode is Complete

SPARK-8360 Make continuous Parquet writes consistent with non-continuous Parquet writes

SPARK-8360 Add DataFrameWriter.foreach to allow the user consuming data in ContinuousQuery

SPARK-8360 Add ContinuousQueryInfo to make ContinuousQueryListener events serializable

SPARK-8360 Add support for complete output mode

SPARK-8360 Disable schema inference for streaming datasets on file streams

SPARK-8360 Properly explain the streaming queries

SPARK-8360 Disable support for multiple streaming aggregations

SPARK-8360 Add ConsoleSink for structure streaming to display the dataframe on the fly

SPARK-8360 Add support for batch jobs correctly inferring partitions from data written with file stream sink

SPARK-8360 StreamExecution.awaitOffset may take too long because of thread starvation

SPARK-8360 Fix flaky test: o.a.s.sql.util.ContinuousQueryListenerSuite "event ordering"

SPARK-8360 Add support in file stream source for reading new files added to subdirs

SPARK-8360 Refactor StreamTests to test for source fault-tolerance correctly.

SPARK-8360 Refactor DataSource to ensure schema is inferred only once when creating a file stream

SPARK-8360 Add partitioned parquet support file stream sink

SPARK-8360 Add a file sink log to support versioning and compaction

SPARK-8360 Fix a race condition in StreamExecution.processAllAvailable

SPARK-8360 Python API for methods introduced for Structured Streaming

SPARK-8360 Fix the race conditions in MemoryStream and MemorySink

SPARK-8360 Move FileSource offset log into checkpointLocation

SPARK-8360 Define analysis rules for operations not supported in streaming

SPARK-8360 Add a note to warn that onQueryProgress is asynchronous

SPARK-8360 QueryProgress should be post after committedOffsets is updated

SPARK-8360 StateStoreCoordinator should extend ThreadSafeRpcEndpoint

SPARK-8360 Memory Sink

SPARK-8360 Method to determine if Dataset is bounded or not

SPARK-8360 Allow multiple continuous queries to be started from the same DataFrame

SPARK-8360 Streaming Aggregation

SPARK-8360 Add processing time trigger

SPARK-8360 Windowing for structured streaming

SPARK-8360 Add a workaround for HADOOP-10622 to fix DataFrameReaderWriterSuite

SPARK-8360 Simple FileSink for Parquet

SPARK-8360 WAL for determistic batches with IDs

SPARK-8360 Old streaming DataFrame proposal by Cheng Hao (Intel)

SPARK-8360 State Store: A new framework for state management for computing Streaming Aggregates

SPARK-8360 Rename DataFrameWriter.stream DataFrameWriter.startStream

SPARK-8360 Add MetadataLog and HDFSMetadataLog

SPARK-8360 ContinuousQueryManagerSuite floods the logs with garbage

SPARK-8360 Flaky test: o.a.s.sql.util.ContinuousQueryListenerSuite.event ordering

SPARK-8360 Remove DataStreamReader/Writer

SPARK-8360 Add FileStreamSource

SPARK-8360 API for managing streaming dataframes

SPARK-8360 API design: external state management

SPARK-8360 Initial infrastructure

SPARK-8360 API design: convergence of batch and streaming DataFrame