[SPARK-43343] Spark Streaming is not able to read a .txt file whose name has [] special character

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 02/May/23 18:28
• Updated: 09/May/23 03:55
• Resolved: 09/May/23 03:55

Description

• For example, If a directory contains a following file:
  /path/abc[123]
  and users would load spark.readStream.format("text").load("/path") as stream input. It throws an exception, saying no matching path /path/abc[123]. Spark thinks abc[123] is a regex that only matches file named abc1, abc2 and abc3.

• Upon investigation this is due to how we getBatch in the FileStreamSource. In `FileStreamSource` we already check file pattern matching and find all match file names. However, in DataSource we check for glob characters again and try to expend it here.