Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38234

Provide monitoring REST API for Structured Streaming



    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • Structured Streaming
    • None


      In SPARK-38234 {}Structured Streaming is added to the history server and a "Structured Streaming" tab appears in the history UI when a streaming query is present. However, even though a store exists for it and the data is presented in the UI, this data is not exposed as a REST API. This data can be used for monitoring, detecting streaming and to build custom dashboards. This monitoring API will be similar to the monitoring APIs that are present for DStreams - refer SPARK-18470.

      In this change, we plan to add two simple APIs that expose the data in the store and can be used to monitor streaming queries. 

      Summary API

      To list the summary of all existing streaming queries.

      GET /{appId}/sql/streamingqueries

      Response is list of StreamingQueryData.

      Progress API

      To list the progress events of a specific streaming query by runId

      User can also specify how many of the most recent events needs to be retrieved by using the last query parameter. By default, we can return the most recent progress event i.e. last is set to 1.

      GET /{appId}/sql/streamingqueries/{runId}/progress?last={N}

      Response is list of StreamingQueryProgress.

      Note: We are not introducing new objects for the response since we are just returning the data from the store without aggregation, these are existing event structures.

      Attached sample I/O and screenshots.{}


        1. StreamingAPI-SS2.jpg
          135 kB
          Karthik Subramanian
        2. StreamingAPI-SS1.jpg
          73 kB
          Karthik Subramanian
        3. StreamingAPIsSampleIO.txt
          1 kB
          Karthik Subramanian



            Unassigned Unassigned
            yeskarthik Karthik Subramanian
            0 Vote for this issue
            3 Start watching this issue