[SPARK-38056] Structured streaming not working in history server when using LevelDB - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.1.2, 3.2.0
Fix Version/s: 3.3.0, 3.1.4, 3.2.2
Component/s: Structured Streaming, Web UI
Labels:
None

Description

In SPARK-31953, structured streaming support is added to history server. However this does not work when spark.history.store.path is set to save app info using LevelDB.

This is because one of the keys of StreamingQueryData, runId, is UUID type, which is not supported by LevelDB. When replaying event log file in history server, StreamingQueryStatusListener will throw an exception when writing info to the store, saying "java.lang.IllegalArgumentException: Type java.util.UUID not allowed as key.".

Example event log is provided in attachments. When opening it in history server with spark.history.store.path set to somewhere, no structured streaming info is available.