Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Incomplete
-
2.2.0, 2.3.0, 2.4.0
-
None
Description
When there are multiple tasks at the same time append a HadoopFsRelation,
there will be an error, there are the following two errors:
1. A task will succeed, but the data will be wrong and more data than excepted will appear
2. Other tasks will fail with java.io.FileNotFoundException: Failed to get file status skip_dir/_temporary/0, ENOENT: No such file or directory
Attachments
Issue Links
- relates to
-
SPARK-24194 HadoopFsRelation cannot overwrite a path that is also being read from
- In Progress
- links to