Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24238

HadoopFsRelation can't append the same table with multi job at the same time.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.2.0, 2.3.0, 2.4.0
    • None
    • SQL

    Description

      When there are multiple tasks at the same time append a HadoopFsRelation,
      there will be an error, there are the following two errors:

      1. A task will succeed, but the data will be wrong and more data than excepted will appear
      2. Other tasks will fail with  java.io.FileNotFoundException: Failed to get file status skip_dir/_temporary/0, ENOENT: No such file or directory

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-24194 HadoopFsRelation cannot overwrite a path that is also being read from

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • In Progress
          links to

          Pull request #21286 [Github] Pull Request #21286 (zheh12)

          Web Link GitHub Pull Request #21286

          Activity

            People

              Unassigned Unassigned
              yangz yangz
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: