[SPARK-35862] Watermark timestamp only can be format in UTC timeZone, unfriendly to users in other time zones

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.1.2
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Yazhi Wang
• Created: 23/Jun/21 10:39
• Updated: 26/Sep/21 00:52
• Resolved: 26/Sep/21 00:52

Description

Timestamp is formatted in `ProgressReporter` by `formatTimestamp` for watermark and eventTime stats. the timestampFormat is hardcoded in UTC time zone.

`

private val timestampFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'") // ISO8601
timestampFormat.setTimeZone(DateTimeUtils.getTimeZone("UTC"))

`

When users set the different timezone by java options `-Duser.timezone` , they may be confused by the information mixed with different timezone.

eg

`

 2021-06-23 16:12:07 [stream execution thread for [id = 92f4f363-df85-48e9-aef9-5ea6f2b70316, runId = 5733ef8e-11d1-46c4-95cc-219bde6e7a20]] INFO [MicroBatchExecution:54]: Streaming query made progress: {
"id" : "92f4f363-df85-48e9-aef9-5ea6f2b70316",
"runId" : "5733ef8e-11d1-46c4-95cc-219bde6e7a20",
"name" : null,
"timestamp" : "2021-06-23T08:11:56.790Z",
"batchId" : 91740,
"numInputRows" : 2577,
"inputRowsPerSecond" : 155.33453887884266,
"processedRowsPerSecond" : 242.29033471229786,
"durationMs" :

{ "addBatch" : 8671, "getBatch" : 3, "getOffset" : 1139, "queryPlanning" : 79, "triggerExecution" : 10636, "walCommit" : 162 }

,
"eventTime" :

{ "avg" : "2021-06-23T08:11:46.307Z", "max" : "2021-06-23T08:11:55.000Z", "min" : "2021-06-23T08:11:37.000Z", "watermark" : "2021-06-23T07:41:39.000Z" }

,

`

maybe we need to unified the timezone for time format

Issue Links

• links to: [Github] Pull Request #33061 (toujours33)
• links to: [Github] Pull Request #33061 (toujours33)