[SPARK-19013] java.util.ConcurrentModificationException when using s3 path as checkpointLocation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Later
• Affects Version/s: 2.0.2
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Tim Chan
• Created: 27/Dec/16 17:40
• Updated: 23/Mar/17 13:51
• Resolved: 23/Mar/17 00:09

Description

I have a structured stream job running on EMR. The job will fail due to this

Multiple HDFSMetadataLog are using s3://mybucket/myapp org.apache.spark.sql.execution.streaming.HDFSMetadataLog.org$apache$spark$sql$execution$streaming$HDFSMetadataLog$$writeBatch(HDFSMetadataLog.scala:162)

There is only one instance of this stream job running.