Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38605

Retrying on file manager operation in HDFSMetadataLog

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 3.4.0
    • None
    • Structured Streaming
    • None

    Description

      Currently HDFSMetadataLog uses CheckpointFileManager to do some file operation like opening metadata file. It is very easy to be affected by network blips and causes the streaming query failed. Although we can restart the streaming query, but it takes more time to recover.

      Such file operations should be resilient with such situation by retrying.

      Attachments

        Activity

          People

            Unassigned Unassigned
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: