[SPARK-27975] ConsoleSink should display alias and options for streaming progress

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jacek Laskowski
• Created: 07/Jun/19 10:00
• Updated: 16/Mar/20 22:50

Description

console sink shows itself in progress with this internal instance representation as follows:

  "sink" : {
    "description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
  }

That is not very user-friendly and would be much better for debugging if it included the alias and options as socket does:

  "sources" : [ {
    "description" : "TextSocketV2[host: localhost, port: 8888]",
    ...
  } ],

The entire sample progress looks as follows:

19/06/07 11:47:18 INFO MicroBatchExecution: Streaming query made progress: {
  "id" : "26bedc9f-076f-4b15-8e17-f09609aaecac",
  "runId" : "8c365e74-7ac9-4fad-bf1b-397eb086661e",
  "name" : "socket-console",
  "timestamp" : "2019-06-07T09:47:18.969Z",
  "batchId" : 2,
  "numInputRows" : 0,
  "inputRowsPerSecond" : 0.0,
  "durationMs" : {
    "getEndOffset" : 0,
    "setOffsetRange" : 0,
    "triggerExecution" : 0
  },
  "stateOperators" : [ ],
  "sources" : [ {
    "description" : "TextSocketV2[host: localhost, port: 8888]",
    "startOffset" : 0,
    "endOffset" : 0,
    "numInputRows" : 0,
    "inputRowsPerSecond" : 0.0
  } ],
  "sink" : {
    "description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
  }
}